Many studies employ multiple measurement instruments such as human raters, observers, judges, or mechanical gauges to record subject data. It is well known that the consistency of these instruments, commonly called rater reliability, limits the extent to which conclusions should be drawn from the observed data. However, the degree to which rater reliability limits conclusions has traditionally been assessed in only subjective manners. In this paper, a method is developed for objectively quantifying the impact of rate reliability on the statistical analysis of data from a commonly used collection scheme. This method allows the inclusion of a reliability index in statistical power calculations and is an invaluable tool in the planning of experiments.