The quality and strength of evidence presented in the individual, included reviews should influence the conclusions drawn in the systematic review of these. The quality and scope of published reviews varies widely. The strength of the conclusions and the ability to provide decision-makers with reliable information depends on the inclusion of reviews that meet a minimum standard of quality. When assessing the quality of the reviews, one should try to avoid being influenced by extraneous variables, such as authors, institutional affiliations and journal names; and should focus on the quality of the conduct of the review. Although the researchers will usually have to do this via an assessment of the quality of report, with the hope that initiatives such as PRISMA (formerly, QUOROM) which assist by facilitating adequate standards of reporting [26 (link)].
The AMSTAR tool [27 (link)], which became available after we started work on our review of reviews, is the only tool that we are aware of that has been validated as a means to assess the methodological quality of systematic reviews and could be used in the review of reviews to determine if the potentially eligible reviews meet minimum requirements based on quality. While the authors of the AMSTAR paper [27 (link)] recognise the need for further testing of the AMSTAR tool, important domains identified within the tool are: establishing the research question and inclusion criteria before the conduct of the review, data extraction by at least two independent data extractors, comprehensive literature review with searching of at least two databases, key word identification, expert consultation and limits applied, detailed list of included/excluded studies and study characteristics, quality assessment of included studies and consideration of quality assessments in analysis and conclusions, appropriate assessment of homogeneity, assessment of publication bias and a statement of any conflict of interest.
Although our review of reviews began before the publication of the AMSTAR tool, we used similar domains to assess review quality. Our assessment criteria are shown below and provide a structure that can be used to report the quality and comparability of the included reviews to help readers assess the strength of the evidence in the review of reviews:
▪ The extent of searching undertaken: Are the databases searched, years searched and restrictions applied in the original review clearly described? Information on the extent of searching should be clearly provided, to allow for a comprehensive assessment of the scope of the review.
▪ Description of review selection and inclusion criteria: Do the authors of the original review provide details of study selection and eligibility criteria and what are these details? This information should be clearly reported in the systematic review of reviews.
▪ Assessment of publication bias: Did the authors of the original review seek additional information from authors of the studies they included? Are there any details of statistical tests (such as funnel plot analysis) to assess for publication bias?
▪ Assessment of heterogeneity: Did the authors of the original review discuss or provide details of any tests of heterogeneity? In the presence of significant heterogeneity, were statistical tests used to address this?
▪ Comparability of included reviews: Are the reviews comparable in terms of eligibility criteria, study characteristics and primary outcome of interest? For example, in our review of reviews on fetal fibronectin and transvaginal cervical ultrasound for predicting preterm birth, [8 (link)] we included reviews that had incorporated studies among women who were both symptomatic and asymptomatic for preterm birth. As a means of addressing comparability of the included reviews, we provided details of the number of women in each group separately and reported the results for each group separately, where applicable.
The AMSTAR tool [27 (link)], which became available after we started work on our review of reviews, is the only tool that we are aware of that has been validated as a means to assess the methodological quality of systematic reviews and could be used in the review of reviews to determine if the potentially eligible reviews meet minimum requirements based on quality. While the authors of the AMSTAR paper [27 (link)] recognise the need for further testing of the AMSTAR tool, important domains identified within the tool are: establishing the research question and inclusion criteria before the conduct of the review, data extraction by at least two independent data extractors, comprehensive literature review with searching of at least two databases, key word identification, expert consultation and limits applied, detailed list of included/excluded studies and study characteristics, quality assessment of included studies and consideration of quality assessments in analysis and conclusions, appropriate assessment of homogeneity, assessment of publication bias and a statement of any conflict of interest.
Although our review of reviews began before the publication of the AMSTAR tool, we used similar domains to assess review quality. Our assessment criteria are shown below and provide a structure that can be used to report the quality and comparability of the included reviews to help readers assess the strength of the evidence in the review of reviews:
▪ The extent of searching undertaken: Are the databases searched, years searched and restrictions applied in the original review clearly described? Information on the extent of searching should be clearly provided, to allow for a comprehensive assessment of the scope of the review.
▪ Description of review selection and inclusion criteria: Do the authors of the original review provide details of study selection and eligibility criteria and what are these details? This information should be clearly reported in the systematic review of reviews.
▪ Assessment of publication bias: Did the authors of the original review seek additional information from authors of the studies they included? Are there any details of statistical tests (such as funnel plot analysis) to assess for publication bias?
▪ Assessment of heterogeneity: Did the authors of the original review discuss or provide details of any tests of heterogeneity? In the presence of significant heterogeneity, were statistical tests used to address this?
▪ Comparability of included reviews: Are the reviews comparable in terms of eligibility criteria, study characteristics and primary outcome of interest? For example, in our review of reviews on fetal fibronectin and transvaginal cervical ultrasound for predicting preterm birth, [8 (link)] we included reviews that had incorporated studies among women who were both symptomatic and asymptomatic for preterm birth. As a means of addressing comparability of the included reviews, we provided details of the number of women in each group separately and reported the results for each group separately, where applicable.
Free full text: Click here
