To evaluate the performance of a state-of-the-art cardiopulmonary resuscitation (CPR) artefact suppression method by assessing to what extent the filtered electrocardiogram (ECG) can be correctly diagnosed by emergency medicine doctors.A total of 819 ECG segments were used. Each segment contained two consecutive 10 s intervals, an artefact free interval and an interval corrupted by CPR artefacts. Each ECG segment was digitally processed to remove CPR artefacts using an adaptive filter. Each ECG segment was split into artefact-free and filtered intervals, randomly reordered for dissociation, and independently offered to four reviewers for rhythm annotation. The rhythm annotations of the artefact-free intervals were considered as the gold standard against which the rhythm annotations of the filtered intervals were evaluated. For the filtered intervals, the rater agreement (κ, Kappa score) with the gold standard, the sensitivity and the specificity were computed individually for each reviewer, and jointly through the majority decision of the pool of reviewers (DPR). These results were also compared to those obtained using a commercial shock advisory algorithm (SAA).The agreement between each reviewer and the gold standard was moderate ranging between κ = 0.41-0.64. The sensitivities and specificities ranged between 64.3-95.0%, and 70.0-95.9%, respectively. The agreement for the DPR was substantial with κ = 0.64 (0.62-0.66), a sensitivity of 90.6%, and a specificity of 85.6%. For the SAA, the agreement was fair with κ = 0.33 (0.31-0.35), a sensitivity of 90.3%, and a specificity of 66.4%.Clinicians outperformed the SAA, but specificities remained below the specifications recommended by the American Heart Association. Visual assessment of the filtered ECG by clinicians is not reliable enough, and varies greatly among clinicians. Results considerably improve by considering the consensus decision of a pool of clinicians.