Seven hundred fifty-two randomly selected charts from seven teaching hospitals were rated by pairs of medical record analysts. The Severity of Illness Index was unreliable with an interrater-agreement rate of 73% (kappa statistic = 0.41), and demonstrated a significant (P less than 0.0001) association with the Adverse Patient Occurrence (APO) Index. This suggests that the Severity of Illness Index is not differentiating severity of illness from quality of care. The fair to poor field reliability stems from underlying instrument subjectivity, lack of clear referent groups, and time pressure. The APO Index was also found to be unreliable (r = 0.33 and range = -0.05-0.58). Greater attention should be directed to improving objective discharge abstract, billing, and laboratory data for measuring patient severity and adverse patient occurrences.