Elsevier

The Lancet

Volume 383, Issue 9912, 11–17 January 2014, Pages 166-175
The Lancet

Series
Increasing value and reducing waste in research design, conduct, and analysis

https://doi.org/10.1016/S0140-6736(13)62227-8Get rights and content

Summary

Correctable weaknesses in the design, conduct, and analysis of biomedical and public health research studies can produce misleading results and waste valuable resources. Small effects can be difficult to distinguish from bias introduced by study design and analyses. An absence of detailed written protocols and poor documentation of research is common. Information obtained might not be useful or important, and statistical precision or power is often too low or used in a misleading way. Insufficient consideration might be given to both previous and continuing studies. Arbitrary choice of analyses and an overemphasis on random extremes might affect the reported findings. Several problems relate to the research workforce, including failure to involve experienced statisticians and methodologists, failure to train clinical researchers and laboratory scientists in research methods and design, and the involvement of stakeholders with conflicts of interest. Inadequate emphasis is placed on recording of research decisions and on reproducibility of research. Finally, reward systems incentivise quantity more than quality, and novelty more than reliability. We propose potential solutions for these problems, including improvements in protocols and documentation, consideration of evidence from studies in progress, standardisation of research efforts, optimisation and training of an experienced and non-conflicted scientific workforce, and reconsideration of scientific reward systems.

Introduction

Design, conduct, and analysis of biomedical and public health research form an interdependent continuum. Some specialties have more efficient mechanisms than others to optimise the design, conduct, and analysis of studies, providing the opportunity for different specialties to learn from successful approaches and avoid common pitfalls. The rapid introduction of new biological measurement methods involving genomes, gene products, biomarkers, and their interactions has promoted novel and complex analysis methods that are incompletely understood by many researchers and might have their own weaknesses. Additionally, biomedical and public health research increasingly interacts with many disciplines, using methods and collaborating with scientists from other sciences, such as economics, operational research, behavioural sciences, and informatics,1 heightening the need for careful study design, conduct, and analysis.

These issues are often related to misuse of statistical methods, which is accentuated by inadequate training in methods. For example, a study2 of reports published in 2001 showed that p values did not correspond to the given test statistics in 38% of articles published in Nature and 25% in the British Medical Journal. Prevalent conflicts of interest can also affect the design, analysis, and interpretation of results. Problems in study design go beyond statistical analysis, and are shown by the poor reproducibility of research. Researchers at Bayer3 could not replicate 43 of 67 oncological and cardiovascular findings reported in academic publications. Researchers at Amgen could not reproduce 47 of 53 landmark oncological findings for potential drug targets.4 The scientific reward system places insufficient emphasis on investigators doing rigorous studies and obtaining reproducible results.

Recommendations

  • 1

    Make publicly available the full protocols, analysis plans or sequence of analytical choices, and raw data for all designed and undertaken biomedical research

    • Monitoring—proportion of reported studies with publicly available (ideally preregistered) protocol and analysis plans, and proportion with raw data and analytical algorithms publicly available within 6 months after publication of a study report

  • 2

    Maximise the effect-to-bias ratio in research through defensible design and conduct standards, a well trained methodological research workforce, continuing professional development, and involvement of non-conflicted stakeholders

    • Monitoring—proportion of publications without conflicts of interest, as attested by declaration statements and then checked by reviewers; the proportion of publications with involvement of scientists who are methodologically well qualified is also important, but difficult to document

  • 3

    Reward (with funding, and academic or other recognition) reproducibility practices and reproducible research, and enable an efficient culture for replication of research

    • Monitoring—proportion of research studies undergoing rigorous independent replication and reproducibility checks, and proportion replicated and reproduced

Problems related to research methodology are intricately linked to the training and composition of the scientific workforce, to the scientific environment, and to the reward system. We discuss the problems and suggest potential solutions from all these perspectives. We provide examples from randomised trials, traditional epidemiology studies, systematic reviews, genetic and molecular epidemiology studies, so-called omics, and animal studies. Further reading for each section is provided in the appendix.

Section snippets

The problem

In research, many effects of interest are fairly small, including those seen in clinical trials and meta-analyses,5 biomarker studies,6 traditional7, 8, 9, 10 and genome11 epidemiology studies, and omics.12 Small effects are difficult to distinguish from biases (information, selection, confounding, etc).8, 13 When effects and biases are potentially of similar magnitude, the validity of any signal is questionable. Design choices can increase the signal, decrease the noise, or both. For example,

Problem 1: poor protocols and designs

The extent to which research is done on the basis of a rudimentary protocol or no protocol at all is unknown, because even when protocols are written, they are often not publicly available. Consequently, researchers might improvise during the conduct of their studies, and place undue emphasis on chance findings. Although some improvisation is unavoidable because of unanticipated events during a study (eg, an unexpectedly high dropout rate, or unpredicted adverse events), changes in the research

Problems

Statistical methods can be complex, and continue to evolve in many specialties, particularly novel ones such as omics. However, statisticians and methodologists are only sporadically involved, often leading to flawed designs and analyses.72 Much flawed and irreproducible work has been published, even when only simple statistical tests are involved. Investigators of one study73 examined the use of Fisher's exact test in 71 articles from six major medical journals. When a statistician was a

Replication and repeatability

In most research specialties, great credit is given to the person who first claims a new discovery, with few accolades given to those who endeavour to replicate findings to assess their scientific validity. Cross-validation of a single dataset might yield inflated results because of biases.78 Replication of findings in new samples is often done by the same researcher who made the original claim; this type of replication might be subject to optimism and allegiance biases, might perpetrate the

Conclusions and recommendations

We have outlined several problems and solutions to reduce waste in the design, conduct, and analysis of research. Not all these solutions are equally relevant or practicable for all research disciplines, and each specialty might need to prioritise which changes are most crucial. For example, panel 2 lists the ten most important priorities for animal research.

To maximise motivation for change, reductions of waste in research will need behavioural changes, not only from researchers, but also from

References (101)

  • VM Montori et al.

    Patient-important outcomes in diabetes—time for consensus

    Lancet

    (2007)
  • DL Veenstra et al.

    A formal risk-benefit framework for genomic tests: facilitating the appropriate translation of genomics into clinical practice

    Genet Med

    (2010)
  • PA Deverka et al.

    Stakeholder assessment of the evidence for cancer genomic tests: insights from three case studies

    Genet Med

    (2012)
  • MJ Khoury et al.

    Population sciences, translational research, and the opportunities and challenges for genomics to reduce the burden of cancer in the 21st century

    Cancer Epidemiol Biomarkers Prev

    (2011)
  • E García-Berthou et al.

    Incongruence between test statistics and P values in medical papers

    BMC Med Res Methodol

    (2004)
  • F Prinz et al.

    Believe it or not: how much can we rely on published data on potential drug targets?

    Nat Rev Drug Discov

    (2011)
  • CG Begley et al.

    Drug development: raise standards for preclinical cancer research

    Nature

    (2012)
  • JP Ioannidis et al.

    Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses

    JAMA

    (2011)
  • S Greenland et al.

    Bias analysis. Chapter 19

  • S Greenland

    Multiple comparisons and association selection in general epidemiology

    Int J Epidemiol

    (2008)
  • JP Ioannidis

    Why most discovered true associations are inflated

    Epidemiology

    (2008)
  • LA Hindorff et al.

    Potential etiologic and functional implications of genome-wide association loci for human diseases and traits

    Proc Natl Acad Sci USA

    (2009)
  • S Senn

    Statistical issues in drug development

    (2007)
  • J Savović et al.

    Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials

    Ann Intern Med

    (2012)
  • JG Lijmer et al.

    Empirical evidence of design-related bias in studies of diagnostic tests

    JAMA

    (1999)
  • AW Rutjes et al.

    Evidence of bias and variation in diagnostic accuracy studies

    CMAJ

    (2006)
  • I Tzoulaki et al.

    Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study

    BMJ

    (2011)
  • JP Ioannidis et al.

    The false-positive to false-negative ratio in epidemiologic studies

    Epidemiology

    (2011)
  • GH Guyatt et al.

    GRADE: an emerging consensus on rating quality of evidence and strength of recommendations

    BMJ

    (2008)
  • MA Hlatky et al.

    Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association

    Circulation

    (2009)
  • JP Ioannidis et al.

    Assessment of cumulative evidence on genetic associations: interim guidelines

    Int J Epidemiol

    (2008)
  • P Boffetta et al.

    Recommendations and proposed guidelines for assessing the cumulative evidence on joint effects of genes and environments on cancer occurrence in humans

    Int J Epidemiol

    (2012)
  • A-W Chan et al.

    Increasing value and reducing waste: addressing inaccessible research

    Lancet

    (2014)
  • A Brazma et al.

    Minimum information about a microarray experiment (MIAME)–toward standards for microarray data

    Nat Genet

    (2001)
  • I Simera et al.

    A catalogue of reporting guidelines for health research

    Eur J Clin Invest

    (2010)
  • CR Hooijmans et al.

    A gold standard publication checklist to improve the quality of animal studies, to fully integrate the three Rs, and to make systematic reviews more feasible

    Altern Lab Anim

    (2010)
  • C Kilkenny et al.

    Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research

    PLoS Biol

    (2010)
  • K Dwan et al.

    Comparison of protocols and registry entries to published reports for randomised controlled trials

    Cochrane Database Syst Rev

    (2011)
  • SD Halpern et al.

    The continuing unethical conduct of underpowered clinical trials

    JAMA

    (2002)
  • HI Keen et al.

    The prevalence of underpowered randomized clinical trials in rheumatology

    J Rheumatol

    (2005)
  • SE Maxwell

    The persistence of underpowered studies in psychological research: causes, consequences, and remedies

    Psychol Methods

    (2004)
  • D Moher et al.

    Statistical power, sample size, and their reporting in randomized controlled trials

    JAMA

    (1994)
  • MO Soares et al.

    An evaluation of the feasibility, cost and value of information of a multicentre randomised controlled trial of intravenous immunoglobulin for sepsis (severe sepsis and septic shock): incorporating a systematic review, meta-analysis and value of information analysis

    Health Technol Assess

    (2012)
  • A Ocana et al.

    When are “positive” clinical trials in oncology truly positive?

    J Natl Cancer Inst

    (2011)
  • A Qaseem et al.

    Current pharmacologic treatment of dementia: a clinical practice guideline from the American College of Physicians and the American Academy of Family Physicians

    Ann Intern Med

    (2008)
  • DS Kazi et al.

    Repeat revascularization is a faulty end point for clinical trials

    Circ Cardiovasc Qual Outcomes

    (2012)
  • I Ferreira-González et al.

    Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials

    BMJ

    (2007)
  • KA Robinson et al.

    A systematic examination of the citation of prior research in reports of randomized, controlled trials

    Ann Intern Med

    (2011)
  • I Chalmers et al.

    How to increase value and reduce waste when research priorities are set

    Lancet

    (2014)
  • JP Ioannidis et al.

    The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence

    BMJ

    (2010)
  • Cited by (1059)

    View all citing articles on Scopus
    View full text