PurposeThe comparison of large volumes of complex data resulting from numerical modelling in computational electromagnetics is a demanding task, especially when validating the performance of numerical models against experimental results and testing experimental repeatability. “By‐eye” comparisons can lead to inconsistencies and inherent subjectivity. This paper establishes a “visual” benchmark by which comparisons can be made and therefore used to assist in the development of an algorithmic approach to data comparison.Design/methodology/approachThis new method presented here is based on the Cooper‐Harper Rating Scale, which is a test pilot's evaluation‐rating instrument. This has been modified through qualitative research. The assertion that the rating scale will leave the group mean response unaltered but will reduce the variance has been statistically tested.FindingsThe proposed rating scale provides a calibration technique by which to benchmark comparisons. The scale also reduces subjectivity by producing an overall quantitative measure of similarity. The paper concludes with an application of the rating scale to assessment of a candidate algorithmic approach against correlation.Research limitations/implicationsThe research findings are based on small data sets, which is a limit imposed by the industrial environment in which this scale will be used.Practical implicationsThis paper provides a tool to overcome some of the key substantial difficulties in communicating similarity or difference, namely that “similarity” and “difference” have no stand‐alone definition, there is a lack of a shared language for the comparisons and little commonality for a decision‐making framework.Originality/valueThis paper provides modellers and experimentalists in computational electromagnetics (particularly electromagnetic compatibility) with a structured approach to quantifying the quality of comparative results.