Original ArticleA checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus
Introduction
Randomized controlled trials (RCTs) are widely accepted as the most reliable method to assess the efficacy of treatments. Assessing the effectiveness of nonpharmacological treatments (NPTs) such as surgery, use of technical devices, behavioral therapy, or psychotherapy presents specific methodological issues [1], [2]. In NPT trials, it is often difficult to perform a sham intervention, and the blinding of participants and care providers is frequently impossible [1], [2], [3], [4]. Moreover, contrary to pharmacological treatment, in NPT, the success of the treatment often depends on the skill and experience of care providers [5], [6], [7], [8], [9], [10], [11]. Finally, NPTs usually involve several components and are difficult to standardize.
Assessing the quality of reports of trials is particularly important for clinicians' critical appraisal of healthcare literature and for systematic reviews [12], [13], [14]. Methodological issues such as lack of adequate allocation concealment or double blinding can influence treatment effect estimates [15], [16] and, if the raw material is flawed, then the conclusions of systematic reviews are more likely to compound these biases. The QUOROM statement [17] specifies that “the criteria and process used for validity assessment” should be reported.
Several tools can be used to assess the quality of reports [18], [19]. They are heterogeneous, and to our knowledge only a few were developed with appropriate rigor and involved an external literature review and a consensus method with experienced methodologists [20], [21]. Moreover, these instruments have been developed and validated in the context of pharmacological treatments. They do not take into account specific methodological issues in assessing NPTs, such as influence of care providers [22], [23], [24], standardization, feasibility of blinding, and the risk of bias in unblinded trials. As such, these assessment tools may not be appropriate for assessing the quality of an increasing number of reports of NPT trials; it therefore seems important to develop a specific tool to assess the quality of reports of these studies.
We report the development of such an assessment instrument here.
Section snippets
Methods
We applied general methods for instrument development used by others [20], [21], [25], [26]. This process consists of three main steps: definition of the aim of the tool, generating items, and selecting items.
Panel of participants
Of the 154 experts invited, 55 (36%) participated in the survey; their names are listed here in the Acknowledgments section. Other experts did not participate because of lack of time (n = 19) or for other reasons (n = 2), or because they did not respond (n = 78). Nonrespondents were from the Cochrane Collaboration (n = 20), clinical researchers (n = 35), and clinicians (n = 44). The characteristics of the panel of participants are described in Table 1. The participants were experienced in
Discussion
This work resulted in a checklist of items measuring the quality of RCTs assessing NPT. This checklist is simple and quick (i.e., about 10 minutes to complete the checklist) to complete and can be used for critical appraisal of the medical literature, the planning of RCTs assessing NPT and quality assessment of reports of NPT trials included in systematic reviews. A detailed user's guide to help assessors evaluate each item accompanies the checklist.
According to the scientific standard used to
Conclusions
To our knowledge, this is the first checklist developed using the Delphi technique with experienced participants that assesses the quality of reports of trials evaluating NPT. This checklist is simple and quick to complete and should help clinicians in critically appraising the medical literature, reviewers in assessing the quality of reports included in systematic reviews, and researchers in planning clinical trials of NPT. We intend to do secondary validation of the checklist with trial
Acknowledgments
The authors are very grateful to the following experts for their participation in this survey:
D.G. Altman, H.K. Andersen, E. Balk, A. Beamond, S. Bell-Syer, M. Bhandari, M. Boers, H. Boter, R. Bourne, G. Chene, R. Churchill, K.S. Courneya, R. de Bie, C. del Mar, P.J. Devereaux, P. Dieppe, E. Ernst, F. Fairman, M. Federico, D. Fergusson, C. Garritty, P. Glasziou, P.C. Gøtzsche, L. Hartling, S. Hopewell, J. Houghton, M. Judd, P. Jüni, P. Knipschild, T. Kober, T. Koivisto, T. Lasserson, J. Lau, V.
References (38)
- et al.
Blinding was judged more difficult to achieve and maintain in nonpharmacological than pharmacological trials
J Clin Epidemiol
(2004) - et al.
Relationship of surgical volume to short-term mortality, morbidity, and hospital charges in arthroplasty
J Arthroplasty
(1995) - et al.
The effect of centralization of primary surgery on survival in ovarian cancer patients
Obstet Gynecol
(2003) - et al.
The art of quality assessment of RCTs included in systematic reviews
J Clin Epidemiol
(2001) - et al.
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?
Lancet
(1998) - et al.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses
Lancet
(1999) - et al.
Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists
Control Clin Trials
(1995) - et al.
Assessing the quality of reports of randomized clinical trials: is blinding necessary?
Control Clin Trials
(1996) - et al.
The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus
J Clin Epidemiol
(1998) - et al.
Explanatory and pragmatic attitudes in therapeutical trials
J Chronic Dis
(1967)
Randomized controlled trials in surgery: issues and problems
Surgery
Why we need observational studies to evaluate the effectiveness of health care
BMJ
Randomised trials in surgery: problems and possible solutions
BMJ
Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis
JAMA
Relationship between the volume of total hip replacements performed by providers and the rates of postoperative complications in the state of Washington
J Bone Joint Surg Am
The fall and rise of carotid endarterectomy in the United States and Canada
N Engl J Med
Benefit of carotid endarterectomy in patients with symptomatic moderate or severe stenosis
N Engl J Med
Relationship between provider volume and mortality for carotid endarterectomies in New York state
Stroke
Relation between operator and hospital volume and outcomes following percutaneous coronary interventions in the era of the coronary stent
JAMA
Cited by (257)
Comparison of self-management and spa therapy for upper-extremity musculoskeletal disorders: A randomized controlled trial
2024, Annals of Physical and Rehabilitation MedicineScaffold- and graft-based biological augmentation of rotator cuff repair: an updated systematic review and meta-analysis of preclinical and clinical studies for 2010-2022
2023, Journal of Shoulder and Elbow SurgeryDesigning clinical trials of surgical interventions
2023, Handbook for Designing and Conducting Clinical and Translational SurgeryBalneotherapy in spondyloarthropathy: A systematic review
2022, TherapiesSham Surgery Studies in Orthopaedic Surgery May Just Be a Sham: A Systematic Review of Randomized Placebo-Controlled Trials
2020, Arthroscopy - Journal of Arthroscopic and Related Surgery