Systematic reviews are used to inform practice, and develop guidelines and protocols. A questionnaire to quantify the risk of bias in systematic reviews, the review paper assessment (RPA) tool, was developed and tested. A search of electronic databases provided a data set of review articles that were then independently reviewed by two assessors using the RPA. The inter-rater reliability was between moderate and good (κ scores 0.46-0.95). Many reviews did not describe the purpose in terms of population, intervention, comparator and outcome measure (i.e. PICO format), making inter-rater agreement on this question difficult. The RPA discriminated between high-quality reviews and those with a risk of bias (e.g. inadequate reporting of search terms, lack of independent reviewing or inclusion of non-randomized-controlled trials). The RPA questionnaire was revised to ensure that questions (on the basis of clarity of purpose, extent of search, independence of reviewers, randomized-controlled trial inclusion and availability of data) had dichotomous answers so that the positive responses scored one. The risk of bias increases as the score reduces.