A standardized dyslexia screening test can help in identifying the vast number of undiagnosed dyslexics in Indian schools; needless to say, such a test should produce consistent and reproducible results. This study investigated reliability and consistency among raters of a Malayalam-English dyslexia screening in India. Paper-based tests were administered to groups of students, and four raters evaluated the answer sheets of 208 second-grade students (ages 6–7). Inter-rater agreement, intraclass agreement, and internal consistency were calculated. Our findings include good agreement among raters' appraisals for most error types and tasks. Internal consistency for a few tasks was low, possibly because these tasks evaluated more than one skill. A few error types need to be redefined and a few tasks need to be more skill-specific to enable unambiguous and fruitful interpretation by different raters in the future.