PHYSICIANS SEEKING THE BEST INFORMATION ABOUT PARticular interventions often turn to the results of metaanalyses. Meta-analyses, if done correctly according to explicit rules, will include all relevant studies that meet specified criteria, even those unpublished, to produce an unbiased estimate of the intervention’s worth. If the quality of the component studies of a meta-analysis is poor, then a precise summary of those poor studies is unjustified. Since poor-quality studies sometimes produce systematically different results, for example, larger treatment effects, a metaanalysis may not only be deceptively precise, but may yield misleading results. In an attempt to deal directly with issues of study quality, many meta-analyses of therapeutic issues restrict consideration to randomized controlled trials. Experience has shown, however, that even randomized controlled trials sometimes show bias. What has become something of an issue among meta-analysts, is how best to measure study quality and to account for it in performing meta-analyses. Even within randomized controlled trials, quality is an elusive metric. Jadad and colleagues define quality as “the likelihood of the trial design to generate unbiased results” but as Verhagen et al point out in an article that displays the difficulty in getting agreement on what constitutes quality, this only covers internal validity. A complete definition of quality also should take into account the trial’s external validity and its statistical analysis, as well as, perhaps, its ethical aspects. Why would physicians be concerned about evaluating trial quality as part of a meta-analysis? Physicians would be most interested in knowing whether high-quality trials give different summary estimates of treatment effect than lowquality trials. If high-quality trials do give systematically different estimates of treatment effect, then physicians would want to draw clinical conclusions on the basis of highquality not low-quality trials. This could be done using stratification on quality. Systems for assessing quality are of 2 general kinds. There are those that simply assess the presence or absence in the report of a trial of a few key components, such as adequate concealment of allocation. Others use scales in which various items (for example, descriptions of randomization or of dropouts) are given numerical scores that are then totaled. In this issue of THE JOURNAL, 2 complementary articles address these clinical concerns. Juni and colleagues examine trials comparing low-molecular weight heparin with standard heparin for thromboprophylaxis in general surgery. The authors show that whether a trial is called high or low quality depends on which of the 25 scoring scales is used. Consequently, in a meta-analysis, summarizing the high-quality trials separately from the low-quality trials leads to drastically different clinical conclusions, depending again on which scale is used. The figure that Juni and colleagues present is truly astounding. It shows that the difference in treatment effects between purportedly high-quality and lowquality trials changes drastically in magnitude and even may change in direction, depending on the choice of quality scale. Despite seemingly dramatic differences between highquality and low-quality scores, there were no statistically significant associations between individual quality scores and treatment effect. At worst, it seems that scales (with more statistical power) are useless. At best, these scales would show no significant relation to effect size. Particular features of studies, however, do relate to effect size. If such evaluations, applied to other meta-analyses, are found to depend as heavily on the choice of quality scales as Juni et al found in this meta-analysis, then readers cannot rely on quality scales to reach clinical conclusions. However, as Juni et al point out, it is not surprising that the scales give such different results, as they focus on different aspects of trials. Some focus more on aspects of reporting of trials than on the design of the trials. Good quality reporting helps the reader evaluate the quality of the design. Even when certain aspects of trial design may be deficient methodologically, thorough reporting allows the reader to put the trial results in perspective. In attempting to draw clinical conclusions, though, it is necessary to have trials in which the potential for bias has been reduced. If the results of Juni et al are confirmed, it follows that scales should be abandoned, and that readers should concentrate on key components of design, rather than reporting.