Abstract Fully automated Vickers measurement systems with image analysis have been commonly used for over 20 years to measure case depth in carburizing, nitriding, and surface hardening processes. One of the most frequently discussed issues is the reproducibility of results both within individual systems and between different systems. This paper presents examples from industrial users to illustrate the ongoing challenges with these systems and the main points of discussion. The examples highlight the difficulties that arise even when systems operate within the accepted accuracy range of existing standards. Additionally, the paper will discuss future challenges and potential improvements for automatic hardness test systems.