Original Article
A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus

https://doi.org/10.1016/j.jclinepi.2005.05.004Get rights and content

Abstract

Background and Objective

To develop a checklist of items measuring the quality of reports of randomized clinical trials (RCTs) assessing nonpharmacological treatments (NPTs).

Study Design and Setting

The Delphi consensus method was used to select and reduce the number of items in the checklist. A total of 154 individuals were invited to participate: epidemiologists and statisticians involved in the field of methodology of RCTs (n = 55), members of the Cochrane Collaboration (n = 41), and clinicians involved in planning NPT clinical trials (n = 58). Participants ranked on a 10-point Likert scale whether an item should be included in the checklist.

Results

Fifty-five experts (36%) participated in the survey. They were experienced in systematic reviews (68% were involved in the Cochrane Collaboration) and in planning RCTs (76%). Three rounds of the Delphi method were conducted to achieve consensus. The final checklist contains 10 items and 5 subitems, with items related to the standardization of the intervention, care provider influence, and additional measures to minimize the potential bias from lack of blinding of participants, care providers, and outcome assessors.

Conclusions

This tool can be used to critically appraise the medical literature, design NPT studies, and assess the quality of trial reports included in systematic reviews.

Introduction

Randomized controlled trials (RCTs) are widely accepted as the most reliable method to assess the efficacy of treatments. Assessing the effectiveness of nonpharmacological treatments (NPTs) such as surgery, use of technical devices, behavioral therapy, or psychotherapy presents specific methodological issues [1], [2]. In NPT trials, it is often difficult to perform a sham intervention, and the blinding of participants and care providers is frequently impossible [1], [2], [3], [4]. Moreover, contrary to pharmacological treatment, in NPT, the success of the treatment often depends on the skill and experience of care providers [5], [6], [7], [8], [9], [10], [11]. Finally, NPTs usually involve several components and are difficult to standardize.

Assessing the quality of reports of trials is particularly important for clinicians' critical appraisal of healthcare literature and for systematic reviews [12], [13], [14]. Methodological issues such as lack of adequate allocation concealment or double blinding can influence treatment effect estimates [15], [16] and, if the raw material is flawed, then the conclusions of systematic reviews are more likely to compound these biases. The QUOROM statement [17] specifies that “the criteria and process used for validity assessment” should be reported.

Several tools can be used to assess the quality of reports [18], [19]. They are heterogeneous, and to our knowledge only a few were developed with appropriate rigor and involved an external literature review and a consensus method with experienced methodologists [20], [21]. Moreover, these instruments have been developed and validated in the context of pharmacological treatments. They do not take into account specific methodological issues in assessing NPTs, such as influence of care providers [22], [23], [24], standardization, feasibility of blinding, and the risk of bias in unblinded trials. As such, these assessment tools may not be appropriate for assessing the quality of an increasing number of reports of NPT trials; it therefore seems important to develop a specific tool to assess the quality of reports of these studies.

We report the development of such an assessment instrument here.

Section snippets

Methods

We applied general methods for instrument development used by others [20], [21], [25], [26]. This process consists of three main steps: definition of the aim of the tool, generating items, and selecting items.

Panel of participants

Of the 154 experts invited, 55 (36%) participated in the survey; their names are listed here in the Acknowledgments section. Other experts did not participate because of lack of time (n = 19) or for other reasons (n = 2), or because they did not respond (n = 78). Nonrespondents were from the Cochrane Collaboration (n = 20), clinical researchers (n = 35), and clinicians (n = 44). The characteristics of the panel of participants are described in Table 1. The participants were experienced in

Discussion

This work resulted in a checklist of items measuring the quality of RCTs assessing NPT. This checklist is simple and quick (i.e., about 10 minutes to complete the checklist) to complete and can be used for critical appraisal of the medical literature, the planning of RCTs assessing NPT and quality assessment of reports of NPT trials included in systematic reviews. A detailed user's guide to help assessors evaluate each item accompanies the checklist.

According to the scientific standard used to

Conclusions

To our knowledge, this is the first checklist developed using the Delphi technique with experienced participants that assesses the quality of reports of trials evaluating NPT. This checklist is simple and quick to complete and should help clinicians in critically appraising the medical literature, reviewers in assessing the quality of reports included in systematic reviews, and researchers in planning clinical trials of NPT. We intend to do secondary validation of the checklist with trial

Acknowledgments

The authors are very grateful to the following experts for their participation in this survey:

D.G. Altman, H.K. Andersen, E. Balk, A. Beamond, S. Bell-Syer, M. Bhandari, M. Boers, H. Boter, R. Bourne, G. Chene, R. Churchill, K.S. Courneya, R. de Bie, C. del Mar, P.J. Devereaux, P. Dieppe, E. Ernst, F. Fairman, M. Federico, D. Fergusson, C. Garritty, P. Glasziou, P.C. Gøtzsche, L. Hartling, S. Hopewell, J. Houghton, M. Judd, P. Jüni, P. Knipschild, T. Kober, T. Koivisto, T. Lasserson, J. Lau, V.

References (38)

  • R.S. McLeod et al.

    Randomized controlled trials in surgery: issues and problems

    Surgery

    (1996)
  • N. Black

    Why we need observational studies to evaluate the effectiveness of health care

    BMJ

    (1996)
  • P. McCulloch et al.

    Randomised trials in surgery: problems and possible solutions

    BMJ

    (2002)
  • I. Boutron et al.

    Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis

    JAMA

    (2003)
  • H.J. Kreder et al.

    Relationship between the volume of total hip replacements performed by providers and the rates of postoperative complications in the state of Washington

    J Bone Joint Surg Am

    (1997)
  • J.V. Tu et al.

    The fall and rise of carotid endarterectomy in the United States and Canada

    N Engl J Med

    (1998)
  • H.J. Barnett et al.

    Benefit of carotid endarterectomy in patients with symptomatic moderate or severe stenosis

    N Engl J Med

    (1998)
  • E.L. Hannan et al.

    Relationship between provider volume and mortality for carotid endarterectomies in New York state

    Stroke

    (1998)
  • P.D. McGrath et al.

    Relation between operator and hospital volume and outcomes following percutaneous coronary interventions in the era of the coronary stent

    JAMA

    (2000)
  • Cited by (257)

    • Designing clinical trials of surgical interventions

      2023, Handbook for Designing and Conducting Clinical and Translational Surgery
    View all citing articles on Scopus
    View full text