ABSTRACTThe hallmark of evidence‐based anatomy (EBA) is the anatomical meta‐analysis (AMA). The Critical Appraisal Tool for Anatomical Meta‐Analysis (CATAM) was recently published to enable users to appraise AMAs quickly and effectively. The tool is valuable for students and clinicians who need to judge the quality of AMAs, which informs clinical decision making and results in better patient care. Subjective measures of the tool's face and content validity have been established, but establishing its reliability provides a more objective measure of the instrument's dependability. This study investigated the interrater reliability (IRR) of the CATAM between novice and expert raters. Three graduate students and three professors (two anatomists and one pharmacist) read the original CATAM paper, and then had a post hoc meeting to discuss scoring with the tool. Three recent AMAs (published between 2017 and 2022) were randomly chosen from PubMed, and all six raters scored the papers blindly. The intraclass correlation coefficient (ICC) statistic was used to calculate the interrater reliability (IRR) between all scores, and then the ICCs between novice and expert scores were compared. Cronbach's alpha (internal consistency) of the CATAM was also calculated (SPSS 25, Armonk, NY). ICC for AMA‐1 was 0.999 (95% CI, 0.997–0.999), p = 0.000, and alpha was 0.999. ICC for AMA‐2 was 0.994 (95% CI, 0.988–0.998), p = 0.000, and alpha was 0.994. ICC for AMA‐3 was 0.998 (95% CI, 0.995–0.999), p = 0.000, and alpha was 0.998. ANOVA showed no significant differences (p > 0.05) in mean ICCs between raters. The CATAM is a robust tool with excellent IRR (ICC > 0.990) and internal consistency (alpha > 0.990). No significant difference in ICC scores between novices and experts suggests the tool does not require prior expert knowledge to be effective. Now that the reliability of the CATAM is established, it can be more widely adopted by students and physicians worldwide to evaluate the quality of AMAs. The CATAM offers widespread applicability, and can be adopted in medical education, journal clubs, and clinical seminars to critically evaluate AMAs.