Original Article
Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis

https://doi.org/10.1016/j.jclinepi.2012.10.005Get rights and content

Abstract

Objective

To compare methodological characteristics of randomized controlled trials (RCTs) published in higher vs. lower impact Core Clinical Journals.

Study Design and Setting

We searched MEDLINE for RCTs published in 2007 in Core Clinical Journals. We randomly sampled 1,140 study reports in a 1:1 ratio in higher (five general medicine journals with the highest total citations in 2007) and lower impact journals.

Results

Four hundred sixty-nine RCTs proved eligible: 219 in higher and 250 in lower impact journals. RCTs in higher vs. lower impact journals had larger sample sizes (median, 285 vs. 39), were more likely to receive industry funding (53% vs. 28%), declare concealment of allocation (66% vs. 36%), declare blinding of health care providers (53% vs. 41%) and outcome adjudicators (72% vs. 54%), report a patient-important primary outcome (69% vs. 50%), report subgroup analyses (64% vs. 26%), prespecify subgroup hypotheses (42% vs. 20%), and report a test for interaction (54% vs. 27%); P < 0.05 for all differences.

Conclusion

RCTs published in higher impact journals were more likely to report methodological safeguards against bias and patient-important outcomes than those published in lower impact journals. However, sufficient limitations remain such that publication in a higher impact journal does not ensure low risk of bias.

Introduction

Rigorously designed and conducted randomized controlled trials (RCTs) provide high-quality evidence regarding the effects of health care interventions [1]. However, RCTs may have limitations in study design, conduct, analysis, and/or reporting, which potentially reduce confidence in their results. These characteristics may differ in higher vs. lower impact journals, and although article citation counts are known to be influenced by many factors [2], [3], including the database used to obtain citation counts [4], many readers may interpret studies in higher impact journals as providing more trustworthy results [5].

Several studies have examined the association between RCTs' methodological characteristics and journal impact factor (See Appendix A at www.jclinepi.com) [6], [7], [8], [9], [10], [11], [12], [13]. Investigators have found that articles published in higher impact factor journals enrolled more patients, more often concealed treatment allocation and adequately generated allocation sequence [6], and were more likely to report a sample size or power calculation and describe the statistical analysis in more detail [7]. The results regarding the frequency of significant outcome and overall methodological quality varied (Appendix A at www.jclinepi.com) [8], [9], [10], [11], [12], [13].

Seven of the previous studies, however, focused on specialty fields, and all studies had narrow inclusion criteria of journals for investigation. Whether the findings apply to a broader range of higher and lower impact journals remain uncertain. We therefore examined the design, conduct, analysis, and methodological quality of a large cohort of RCTs [14] in the set of leading Core Clinical Journals, as defined by the National Library of Medicine [15]. The objective of our study was to compare characteristics of RCTs in higher vs. lower impact journals. We hypothesized that studies published in higher impact journals may use more rigorous methods in the design, conduct, analysis, and reporting of RCTs.

Section snippets

Methods

Our study protocol, describing methods of selection of articles and abstraction of data but focusing on subgroup analyses and not addressing the questions in this article, was published previously [16]. We briefly summarize the methods here.

Results

We included 469 studies (Fig. 1), of which 219 were published in higher impact journals and 250 in lower impact journals. The interrater agreement for study eligibility was 95% (κ = 0.80) and 91% (κ = 0.82) for reporting of subgroup analyses.

RCTs in higher impact journals were more likely to have larger sample sizes and report industry funding (Table 1). We also found a statistically significant association between industry funding (partial or full) and sample size (median, 398 patient per arm

Discussion

We found that, on average, higher impact Core Clinical Journals reported larger trials. These trials more frequently included a patient-important primary outcome, concealed treatment allocation, applied blinding, prespecified subgroup analyses, and tested for interaction (Table 1, Table 2, Table 3, Table 4). However, a substantial proportion of trials in higher impact journals also suffered from important methodological limitations. Over a quarter of those studies did not report concealment of

Acknowledgment

The authors thank Monica Owen for administrative assistance. They also thank Aravin Duraikannan for developing the electronic study forms.

References (37)

  • P. Dreyfuss et al.

    In response to treatment of neck pain

    Eur Spine J

    (2008)
  • L.L. Gluud et al.

    The journal impact factor as a predictor of trial quality and outcomes: cohort study of hepatobiliary randomized clinical trials

    Am J Gastroenterol

    (2005)
  • L.M. Kuroki et al.

    Methodology and analytic techniques used in clinical research: associations with journal impact factor

    Obstet Gynecol

    (2009)
  • Y. Littner et al.

    Negative results and impact factor: a lesson from neonatology

    Arch Pediatr Adolesc Med

    (2005)
  • Z. Kanaan et al.

    The value of lesser-impact-factor surgical journals as a source of negative and inconclusive outcomes reporting

    Ann Surg

    (2011)
  • C. Barbui et al.

    Validity of the impact factor of journals as a measure of randomized controlled trial quality

    J Clin Psychiatry

    (2006)
  • K.P. Lee et al.

    Association of journal quality indicators with methodological quality of clinical research articles

    JAMA

    (2002)
  • X. Sun et al.

    The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review

    BMJ

    (2011)
  • Cited by (0)

    Competing interest: All authors declare that they have no competing interests.

    Funding/support: M.B. is supported by santésuisse and the Gottfried and Julia Bangerter-Rhyner Foundation. J.W.B is funded by a New Investigator Award from the Canadian Institutes of Health Research and the Canadian Chiropractic Research Foundation. D.B. is supported by the European Union (grant award health-F5-2009-223060). D.M. is supported by a research scholarship from the Swiss National Science Foundation (PBBSP3-124436 and PASMP3-132571) and Lichtenstein-Stiftung, Basel, Switzerland. P.D. is supported by a Dennis W. Jahnigan Carreer Development Award by the American Geriatrics Society. S.K.S. holds a Rudy Falk Clinician Scientist Award. P.A.-C. is funded by a Miguel Servet research contract from the Instituto de Salud Carlos III (CP09/00137). J.J.Y. is supported by a Hamilton Health Sciences Early Career Award.

    Contribution: Study conceptualization: X.S. and G.H.G.; study design: X.S., G.H.G., M.B., J.W.B., E.A.A, S.D.W, and D.G.A; acquisition of data: M.B., E.A.A., J.W.B., N.D.-G., J.J.Y., F.M., M.M.B., D.B., D.M., P.O.V., G.M., S.K.S., P.D., B.C.J., P.A.-C., B.H., X.S., Truong, Dattani, and Bhatnagar; analysis and interpretation of data: X.S., G.H.G., and S.D.W.; draft of the manuscript: M.M.B., E.A.A., X.S., and G.H.G.; critical revision of the manuscript: X.S., J.W.B., M.B., G.H.G., J.J.Y., B.H., D.B., D.M., P.O.V., B.C.J., P.D., S.D.W., D.G.A., G.M., F.M., N.D.-G., P.A.-C., and S.K.S.; and administrative, technical, and material support: X.S.

    Ethical approval: Ethical approval was not required for this study.

    View full text