Review Article
Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported

https://doi.org/10.1016/j.jclinepi.2014.09.022Get rights and content

Abstract

Objectives

To survey the outcomes used in Cochrane Reviews, as part of our work within the Core Outcome Measures in Effectiveness Trials Initiative.

Study Design and Setting

A descriptive survey of Cochrane Reviews, divided by Cochrane Review Group (CRG), published in full for the first time in 2007 and 2011. Outcomes specified in the methods section of each review and outcomes reported in the results section of each review were of interest, in this exploration of the common use of outcomes and core outcome sets (COS).

Results

Seven hundred eighty-eight reviews, specifying 6,127 outcomes, were included. When we excluded specified outcomes from the 86 reviews that did not include any studies, we found that 1,996 (37%) specified outcomes were not reported. Of the 361 new reviews with studies from 2011, 113 (31%) had a “summary of findings” table (SoF). Fifteen broad outcome categories were identified and used to manage the outcome data. We found consistency in the use of these categories across CRGs but inconsistency in outcomes within these categories.

Conclusion

COS have been used rarely in Cochrane Reviews, but the introduction of SoF makes the development and application of COS timelier than ever.

Introduction

Cochrane Reviews, with over 5,500 published in full online by 2014, have been described as “unique because they are both produced by, and are relevant to, everyone interested in the effects of human health care” [1]. The Cochrane Collaboration, celebrating its 21st “birthday” this year, prides itself on preparing, maintaining, and promoting access to high-quality, timely research evidence for health care decision making. It does this by supporting the rigorous conduct and reporting of systematic reviews. However, one of the difficulties often faced by systematic reviewers, when synthesizing the evidence from individual studies, is heterogeneity in the outcomes measured in these studies [2]. This difficulty not only presents when clinical trials on a similar topic or condition use a variety of outcome measures, but also where researchers measure the same outcome in a variety of ways [3]. Adding to this is the problem of selective reporting of outcomes in research reports, which can lead to outcome reporting bias [4], [5], [6]. This bias arises when a selection of the originally recorded outcomes is chosen for reporting in study publications, on the basis of their results [7]. The extent of this was highlighted in a review of 2,562 trials included in 283 Cochrane Reviews [8]. Outcome reporting bias was suspected in at least one trial in 35% of the examined reviews. In a sensitivity analysis of 81 of the included reviews that had a single meta-analysis of the review primary outcome, an assessment of the impact of outcome reporting bias was performed. The analysis demonstrated that 52 of the 81 reviews included at least one trial that had a high suspicion for outcome reporting bias and the treatment effect estimate was reduced by 20% or more in 23% (n = 19) of the reviews [8]. This indicates that outcome reporting bias can present a substantial problem for those who wish to use the findings of Cochrane Reviews, other systematic reviews, and individual trials themselves when making health care decisions.

One way to address the difficulty of outcome variation and outcome reporting bias in systematic reviews is to develop and apply agreed standardized sets of outcomes, known as “core outcome sets” (COS) [2], [3], [4], [9]. For example, published reports on COS are available for asthma in children [10], ulcerative colitis [11], models of maternity care [12], and, most noticeably, rheumatology [13], [14]. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative, launched in 2010, is further advancing efforts for COS development [2], [4], [9]. This Initiative brings researchers interested in COS together to consider methods for COS development and to highlight health care areas in need of COS. The Initiative supports the idea that a COS should represent the minimum to be measured and reported in all clinical trials on a specific condition, while recognizing that outcomes outside the COS might also be important in the context of each study. This use of the COS as a minimum across an entire research area would allow for the results of trials and other studies to be effectively compared, contrasted, and combined, as appropriate [2], [9].

One of the objectives of the COMET Initiative is to develop a strategy to link the development of COS for trials with the specification of outcomes for Cochrane Reviews, including the outcomes in their summary of findings (SoF) table. Developed with the GRADE (Grading of Recommendations Assessment, Development and Evaluation) group, SoF tables have been possible in Cochrane Reviews since 2008. A SoF table presents the main findings of a Cochrane Review in a simple tabular format. These tables provide information on the quality of evidence and on the magnitude of effect of the interventions examined in a review. They allow for the inclusion of up to seven important reported outcomes, providing a way to present the main findings of Cochrane Reviews in a simple and transparent format [15]. They have been shown to help readers understand the results of Cochrane Reviews more correctly and faster and are considered to facilitate a more effective and efficient uptake of key information [16]. However, the SoF table will only be effective for evidence transfer if the outcomes selected for inclusion in the table are appropriate to the review question. To explore these issues as part of our work within COMET, we have performed a survey of Cochrane Reviews to identify the variety of outcome measures used in them.

Section snippets

Aim and objectives

The aim of the study was to survey the outcomes used in Cochrane Reviews. The objectives of the survey were as follows:

  • 1.

    To identify the variety of outcome measures used in Cochrane Reviews.

  • 2.

    To identify and highlight the use of COS in reviews from Cochrane Review Groups (CRG).

  • 3.

    To identify health care areas that might benefit from COS development.

Design

A descriptive survey of Cochrane Reviews, divided by CRG, published in full for the first time in 2007 and 2011, was performed. Newly published 2007 and 2011 reviews were purposively chosen so as to explore any potential change in outcome specifying and reporting over time. Outcomes specified in the methods section of each review and outcomes reported (defined as a reported result on an outcome in the text of the review) in the results section of each review were of interest. We also evaluated

Findings

Three hundred eighty-seven newly published Cochrane Reviews in 2007 from 47 CRGs and 401 newly published Cochrane Reviews in 2011 from 50 CRGs were identified and included. The total number of outcomes specified in the methods sections of all these new reviews was 6,127. When specified outcomes from reviews that had not included any studies were excluded (n = 86 reviews), the proportion of specified outcomes that was reported was 63% (3,367 of 5,363). Of the nonreported specified outcomes, 23%

Discussion

In this survey, outcomes used in the 788 Cochrane Reviews published in full for the first time in 2007 and 2011 were explored. A total of 6,127 specified outcomes were listed in the methods sections of the included reviews. After excluding reviews that did not include any studies, the results demonstrated that 37% of specified outcomes in the newly published Cochrane Reviews were not reported. Legitimate reasons for nonreport in the review (ie, nonmeasurement of the outcome in the included

Conclusion

This survey suggests that there has been minimal use and implementation of COS in Cochrane Reviews. It also reveals, for a large proportion of outcomes, that a reason for not reporting a specified outcome is not provided in the text of a review. This indicates that outcome reporting bias might be a problem for some systematic reviews. With the introduction of SoF tables in Cochrane Reviews, work to develop and apply COS is timelier than ever. Developing and implementing COS is not without its

References (23)

  • J.L. Hutton et al.

    Bias in meta-analysis due to outcome variable selection within studies

    Appl Stat

    (2000)
  • Cited by (0)

    Conflict of interest: One of the authors (V.S.) was a research fellow (part-time, 0.5 full-time equivalent), under the FP7 award, to conduct work for the COMET Initiative. The survey reported here was performed within this employment contract. Although authors V.S., M.C., and P.W. are members of the Cochrane Collaboration, the views expressed in this article are not necessarily those of the Collaboration or the Cochrane groups with which they are associated.

    Funding: The COMET Initiative is pleased to acknowledge funding from the European Union Seventh Framework Programme ([FP7/2007–2013] [FP7/2007–2011]) under grant agreement n°305081.

    View full text