Introduction
What is new?
Key findings- •
There is significant variation in the quality assessment tools used in systematic reviews (SRs) published in the Core Clinical Journals.
- •
A threshold level of study quality is relatively infrequently described as a prerequisite before undertaking meta-analysis.
What this adds to what was known?- •
This cross-sectional analysis highlights the variation in methodology used by authors of interventional, observational, and diagnostic test reviews to grade study quality.
- •
There is a lack of clarity in relation to the specific level of quality necessary for inclusion in SRs.
What is the implication and what should change now?- •
This study provides further evidence of the need for more active or intuitive editorial processes to enhance the reporting of SRs.
- •
Greater awareness, consistency, and acceptance of specific quality assessment tools are needed across SR types.
Systematic reviews (SRs) are engrained within evidence-based medicine being trusted to combine and appraise best available evidence in a robust and consistent manner. SRs should involve “a systematic approach to minimizing biases and random errors which is documented in a materials and methods section” [1]. Bias has been defined as a tendency for results of a primary study to differ from the results expected from a randomized trial, conducted on the same participant group that had no design flaws in its conduct. A key element in the SR process is appraisal of the risk of bias or methodological quality of the constituent primary studies to decide whether their results can be trusted and should contribute to eventual meta-analyses. Failure to make objective decisions concerning the inclusion of various study designs and the threshold level of quality required for a study to be included can introduce bias in the review process; moreover, inclusion of less robust studies risks leading to unreliable, often inflated estimates of treatment effects [2], [3].
In recent years, the Cochrane risk of bias tool has become established in the assessment of risk of bias in randomized designs [4]. However, until now, significant variation has existed in relation to tools used in nonrandomized designs [5]. Sanderson et al. identified 86 tools comprising 53 checklists and 33 scales in an electronic search undertaken in March 2005, with, for example, 23 unique tools developed to assess cohort studies and 19 for case–control studies. Most tools included items related to study variables (86%), design-related bias (86%), and confounding (78%), although certain aspects such as conflict of interest were underrepresented (4%) [5]. Similar results have been identified within SRs concerning the epidemiology of chronic disease with only 55% of reviews referring to quality assessment of primary studies overall [6]. More recently, the Cochrane Collaboration has espoused a new tool designed to appraise the risk of bias inherent in nonrandomized studies [7], although it has yet to find wider application. Similarly, in an assessment of interventional SRs, Moher et al. [8] identified 25 unique tools used in the evaluation of methodological quality of clinical trials. Many of these were found to lack accepted theoretical basis and varied in respect of their size, complexity, and regarding the weight attributed to individual methodological aspects such as randomization procedures or blinding. This assessment predated the development of the Cochrane risk of bias tool [4], however, but was echoed in a more recent analysis of interventional SRs within the field of general health research and physical therapy [9].
Notwithstanding the disagreement in respect of the appropriate tool to assess methodological quality, there is also variation regarding the threshold level of primary study quality or risk of bias above which studies may be included in quantitative synthesis. The Cochrane Collaboration, for example, advocates a domain-based assessment of bias and has suggested that a high risk of bias for any one of the individual domains would render a primary study at high risk of bias overall [4]. Consequently, a binary decision on whether to include a study in a meta-analysis may be made. A method of accounting for variations in study quality is the use of sensitivity analysis, whereby assessment of the overall findings to subjective decisions including inclusion of studies of varying design and quality [10].
The objectives of our study were to assess the frequency with which various quality assessment tools were used within SRs, to investigate whether threshold quality levels were applied when using these tools governing possible inclusion in meta-analysis, and to assess the use of sensitivity analysis based on methodological quality or risk of bias of primary studies within meta-analysis.
