Addressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers

doi:10.1016/j.jclinepi.2013.11.014

Journal of Clinical Epidemiology

Volume 67, Issue 5, May 2014, Pages 560-570

https://doi.org/10.1016/j.jclinepi.2013.11.014 Get rights and content

Abstract

Background

We previously developed an approach to address the impact of missing participant data in meta-analyses of continuous variables in trials that used the same measurement instrument. We extend this approach to meta-analyses including trials that use different instruments to measure the same construct.

Methods

We reviewed the available literature, conducted an iterative consultative process, and developed an approach involving a complete-case analysis complemented by sensitivity analyses that apply a series of increasingly stringent assumptions about results in patients with missing continuous outcome data.

Results

Our approach involves choosing the reference measurement instrument; converting scores from different instruments to the units of the reference instrument; developing four successively more stringent imputation strategies for addressing missing participant data; calculating a pooled mean difference for the complete-case analysis and imputation strategies; calculating the proportion of patients who experienced an important treatment effect; and judging the impact of the imputation strategies on the confidence in the estimate of effect. We applied our approach to an example systematic review of respiratory rehabilitation for chronic obstructive pulmonary disease.

Conclusions

Our extended approach provides quantitative guidance for addressing missing participant data in systematic reviews of trials using different instruments to measure the same construct.

Introduction

What is new?

•
Specific guidance for addressing missing participant data for continuous outcomes measured with different instruments in meta-analyses is currently unavailable.
•
We developed an approach consisting of converting units from different instruments to the units of a reference instrument and applying four increasingly stringent data imputation strategies to address this issue.
•
We provide guidance regarding the importance of and the methods for calculating the proportion of patients who have a clinically important effect and detailed guidance for judging the impact of risk of bias as a result of missing participant data.

Randomized controlled trials (RCTs) often suffer from missing participant data [1]. Missing participant data increase risk of bias in both individual trials and meta-analyses. This is especially concerning in positive trials (ie, those with a significant treatment effect) if, in the intervention group, the outcomes of participants with missing data are worse than the outcomes of those with available data.

The Cochrane Collaboration Handbook has proposed a strategy for handling missing participant data for dichotomous outcomes in systematic reviews. The strategy suggests conducting a complete (available) case analysis complemented by sensitivity analyses of various assumptions regarding outcomes of participants with missing data [2]. One of the most common approaches is to adopt the “worst-case scenario,” in which one assumes that participants with missing data in the intervention group had the worst outcome and those in the control group had the best possible outcome [2]. Although this assumption tests the robustness of the pooled estimates for complete-case analyses, it is typically implausible [1].

Our research group has proposed additional strategies that use a range of plausible assumptions that increasingly challenge the results [1]. These strategies make the assumption that those in the intervention group with missing data do relatively worse than those with available data and those in the control group with missing data do relatively better than those with available data. We have applied these strategies to RCTs [1] and systematic reviews [3] reporting dichotomous outcomes.

Until recently, no methods were available (including the Cochrane Handbook) for addressing missing participant data for continuous outcomes in systematic reviews or for assessing its impact on the confidence in the estimate of effect. We have addressed this gap by extending our work from dichotomous outcomes [3] and proposing an approach involving conducting a complete-case analysis as the primary analysis complemented by sensitivity analyses that apply a series of increasingly stringent assumptions about results in patients with missing continuous outcome data. These sensitivity analyses test the robustness of the results of the primary analysis, allowing reviewers to judge the risk of bias associated with missing participant data in individual trials of the systematic review [4]. This approach is limited to systematic reviews in which all trials used the same measurement instrument. In this article, we extend the approach to systematic reviews pooling trials using different instruments to measure the same construct [eg, dyspnea, fatigue, emotional function, health-related quality of life (HRQoL)]. To further enhance interpretability, we illustrate how to calculate important treatment effects from the derived estimates using the minimally important difference (MID), the smallest difference that patients perceive as important [5], [6] and, in our discussion, show how the results can be interpreted in the context of a clinical practice guideline.

Section snippets

Methods

In developing our initial approach, we reviewed the available literature (including the Cochrane Handbook) on missing participant data [1], [2], [7], [8], [9] and conducted an iterative consultative process involving the nine authors of this article, including clinical epidemiologists, methodologists, and biostatisticians.

Our approach addresses analyses for aggregate trial-level data for conducting a meta-analysis and not analyses of individual participant data meta-analyses or missing

Results

Our proposed approach consists of the following steps:

1.
Choosing the reference measurement instrument,
2.
Converting scores from different instruments to the units of the reference instrument,
3.
Imputing measures of effect and their precision,
4.
Combining observed and imputed data,
5.
Special cases,
6.
Calculating the proportion of patients who have an important treatment effect, and
7.
Judging the impact of missing participant data on quality of evidence (confidence in the complete-case estimate of effect).

Discussion

We developed an approach to address the impact of missing participant data on risk of bias in meta-analyses of continuous variables that use different instruments to measure the same construct. This approach involves first choosing the reference instrument (the instrument with the best combination of an established anchor-based MID, excellent measurement properties, and high familiarity to the target audience) and converting scores to units of the reference instrument. The next step involves a

Acknowledgments

Authors' contributions: S.E. and G.H.G. conceived the study. S.E., E.A.A., R.A.M., X.S., S.D.W., D.H-.A., P.A-.C., B.C.J., and G.H.G. contributed to the study design and developed the approach. S.E. and B.C.J. extracted the outcome estimates, missing data, and missingness mechanisms from the individual studies included in the example systematic review. S.E. applied the approach to the example systematic review. S.E. completed the first draft of the manuscript. S.E. is the first author and

References (41)

S. Ebrahim et al.
Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers
J Clin Epidemiol
(2013)
G.H. Guyatt et al.
GRADE guidelines 6. Rating the quality of evidence-imprecision
J Clin Epidemiol
(2011)
M. Behnke et al.
Home-based exercise is capable of preserving hospital-based improvements in severe chronic obstructive pulmonary disease
Respir Med
(2000)
R.S. Goldstein et al.
Randomised controlled trial of respiratory rehabilitation
Lancet
(1994)
T.L. Griffiths et al.
Results at 1 year of outpatient multidisciplinary pulmonary rehabilitation: a randomised controlled trial
Lancet
(2000)
T. Troosters et al.
Short- and long-term effects of outpatient rehabilitation in patients with chronic obstructive pulmonary disease: a randomized trial
Am J Med
(2000)
J.P. Finnerty et al.
The effectiveness of outpatient pulmonary rehabilitation in chronic lung disease. A randomized controlled trial
Chest
(2001)
T.J. Ringbaek et al.
Rehabilitation of patients with chronic obstructive pulmonary disease. Exercise twice a week is not sufficient!
Respir Med
(2000)
F.R. Lake et al.
Upper-limb and lower-limb exercise training in patients with chronic airflow obstruction
Chest
(1990)
M.A. Puhan et al.
Relative responsiveness of the Chronic Respiratory Questionnaire, St. Georges Respiratory Questionnaire and four other health-related quality of life instruments for patients with chronic lung disease
Respir Med
(2007)

A.L. Ries et al.

Pulmonary rehabilitation: joint ACCP/AACVPR evidence-based clinical practice guidelines

Chest

(2007)

E.A. Akl et al.

LOST to follow-up Information in Trials (LOST–IT): potential impact on estimated treatment effects

BMJ

(2012)

Further issues in meta-analysis: intention to treat issues. The Cochrane Collaboration open learning material

(2012)

E.A. Akl et al.

Addressing dichotomous data for participants excluded from trial analysis: a guide for systematic reviewers

PLoS One

(2013)

H.J. Schunemann et al.

Commentary–goodbye M(C)ID! Hello MID, where do you come from?

Health Serv Res

(2005)

B.C. Johnston et al.

Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units

Health Qual Life Outcomes

(2010)

R.H.H. Groenwold et al.

Dealing with missing outcome data in randomized trials and observational studies

Am J Epidemiol

(2011)

J.P.T. Higgins et al.

Imputation methods for missing outcome data in meta-analysis of clinical trials

Clin Trials

(2008)

Chapter 8: risk of bias

D. Moher et al.

CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials

BMJ

(2010)

Cited by (56)

Comparative efficacy and acceptability of cognitive behavioral therapy delivery formats for insomnia in adults: A systematic review and network meta-analysis
2022, Sleep Medicine Reviews
Citation Excerpt :
When these data were not available, we imputed SDs from other studies included in our network meta-analysis [42]. If different insomnia severity rating scales were used, we chose the standard ISI (score ranges from 0 to 28) as the reference measurement instrument, and we converted mean scores and SDs from other instruments to the units of the reference instrument using the formula described by Ebrahim et al. [43] When studies reported time as hours, we transformed hours into minutes. Appendix Text S2 presents details of the missing SDs imputed for each outcome.
This review compared the efficacy and acceptability of different delivery formats for cognitive behavioral therapy for insomnia (CBT-I) in insomnia. We searched five databases for randomized clinical trials that compared one CBT-I delivery format against another format or control conditions for insomnia in adults. We used pairwise meta-analyses and frequentist network meta-analyses with the random-effects model to synthesize data. A total of 61 unique trials including 11,571 participants compared six CBT-I delivery formats with four control conditions. At post-intervention, with low to high certainty evidence, individual, group, guided self-help, digital assisted, and unguided self-help CBT-I could significantly increase sleep efficiency and total sleep time (TST) and reduce sleep onset latency (SOL), wake after sleep onset (WASO), and insomnia severity compared with treatment as usual (MD range for sleep efficiency: 7.81%–12.45%; MD range for TST: 16.14–33.96 min; MD range for SOL: −22.42 to −13.81 min; MD range for WASO: −40.84 to −19.48 min; MD range for insomnia severity: −6.40 to −3.93) and waitlist (MD range for sleep efficiency: 7.68%–12.32%; MD range for TST: 12.67–30.49 min; MD range for SOL: −19.07 to −10.46 min; MD range for WASO: −47.10 to −19.15 min; MD range for insomnia severity: −7.59 to −5.07). The effects of different CBT-I formats persisted at short-term follow-up (4 wk–6 mo). Individual, group, and digital assisted CBT-I delivery formats would be the more appropriate choices for insomnia in adults, based on post-intervention and short-term effects. Further trials are needed to investigate the long-term effects of different CBT-I formats.
Efficacy and acceptability of psychotherapeutic and pharmacological interventions for trauma-related nightmares: A systematic review and network meta-analysis
2022, Neuroscience and Biobehavioral Reviews
Citation Excerpt :
When exploring the effects of baseline nightmare symptoms, different studies may use different tools to evaluate nightmares. We chose CAPS nightmare items as the reference measurement tool, and converted mean scores and standard deviations from other tools to the units of CAPS nightmare items using the formula developed by Ebrahim et al. (Ebrahim et al., 2014). Furthermore, we performed sensitivity analyses: (1) restricting the analyses only in studies with a low or moderate risk of bias; and (2) excluding the one study that utilized PCT, and separating placebo and waitlist as different nodes in the network graph.
This network meta-analysis compares the efficacy and acceptability of all published psychotherapeutic and pharmacological interventions for trauma-related nightmares (TRN) in adults. The analysis included data from 29 randomized clinical trials involving 14 psychotherapeutic and pharmacological interventions and involved 2214 trauma survivors. Prazosin and image rehearsal therapy (IRT) were found to be the two effective interventions for TRN. Other interventions such as risperidone, paroxetine, cognitive behavioral therapy for insomnia (CBT-I), CBT-I+IRT, prolonged exposure (PE), and IRT+PE, did not show significantly greater efficacy compared with control conditions. The rates of all-cause discontinuations were comparable among majority of the interventions and did not show significant differences compared with control conditions. Prazosin and IRT should be considered as the initial choice of pharmacological and psychotherapeutic interventions for TRN. The efficacy of other pharmacological and psychotherapeutic interventions remains to be demonstrated. Future guidelines and daily clinical decision making on the choice of interventions for TRN should consider these findings.
Addition of corticosteroids to local anaesthetics for chronic non-cancer pain injections: a systematic review and meta-analysis of randomised controlled trials
2020, British Journal of Anaesthesia
Despite common use, the benefit of adding steroids to local anaesthetics (SLA) for chronic non-cancer pain (CNCP) injections is uncertain. We performed a systematic review and meta-analysis of English-language RCTs to assess the benefit and safety of adding steroids to local anaesthetics (LA) for CNCP.
We searched MEDLINE, EMBASE, and CENTRAL databases from inception to May 2019. Trial selection and data extraction were performed in duplicate. Outcomes were guided by the Initiative in Methods, Measurements, and Pain Assessment in Clinical Trials (IMMPACT) statement with pain improvement as the primary outcome and pooled using random effects model and reported as relative risks (RR) or mean differences (MD) with 95% confidence intervals (CIs).
Among 5097 abstracts, 73 trials were eligible. Although SLA increased the rate of success (42 trials, 3592 patients; RR=1.14; 95% CI, 1.03–1.25; number needed to treat [NNT], 13), the effect size decreased by nearly 50% (NNT, 22) with the removal of two intrathecal injection studies. The differences in pain scores with SLA were not clinically meaningful (54 trials, 4416 patients, MD=0.44 units; 95% CI, 0.24–0.65). No differences were observed in other outcomes or adverse events. No subgroup effects were detected based on clinical categories. Meta-regression showed no significant association with steroid dose or length of follow-up and pain relief.
Addition of cortico steroids to local anaesthetic has only small benefits and a potential for harm. Injection of local anaesthetic alone could be therapeutic, beyond being diagnostic. A shared decision based on patient preferences should be considered. If used, one must avoid high doses and series of steroid injections.
PROSPERO #: CRD42015020614.
GRADE guidelines: 17. Assessing the risk of bias associated with missing participant outcome data in a body of evidence
2019, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
Bereitstellung eines GRADE-Leitfadens für die Bewertung des Bias-Risikos eines Evidenzkörpers, wenn in systematischen Übersichtsarbeiten Daten von dichotomen und kontinuierlichen Endpunkten fehlen.
Systematische Bestandsaufnahme der publizierten methodologischen Forschung, iterative Diskussionen, Erprobung in systematischen Übersichtsarbeiten sowie Feedback der GRADE-Arbeitsgruppe.
Der methodische Zugang beginnt mit einer primären Meta-Analyse unter Verwendung einer Analyse der vollständigen Fälle („complete case analysis“), gefolgt von Sensitivitäts-Meta-Analysen, in denen für jede Studie mit fehlenden Daten Imputationen vorgenommen werden und anschließend über alle Studien hinweg gepoolt wird. Für dichotome Endpunkte empfehlen wir die Verwendung des „plausiblen Worst-Case“, in dem die Review-Autorinnen und -autoren davon ausgehen, dass Studien mit fehlenden Daten in den Behandlungsarmen proportional höhere Ereignisraten aufweisen als Studien mit kompletten Daten. Für kontinuierliche Ergebnismaße werden die imputierten Mittelwerte von anderen Studien innerhalb der systematischen Übersichtsarbeit und die Standardabweichungen von der medianen Standardabweichung der Kontrollgruppen aller Studien herangezogen.
Wenn die Ergebnisse der primären Meta-Analyse gegenüber den extremsten Annahmen, die als plausibel angesehen werden, robust sind, stuft man die Vertrauenswürdigkeit der Evidenz aufgrund des Bias-Risikos wegen fehlender Teilnehmer-Ergebnisdaten nicht herab. Wenn sich die Ergebnisse als nicht robust gegenüber plausiblen Annahmen erweisen, würde man die Vertrauenswürdigkeit der Evidenz aufgrund des Bias-Risikos herabstufen.
To provide GRADE guidance for assessing risk of bias across an entire body of evidence consequent on missing data for systematic reviews of both binary and continuous outcomes.
Systematic survey of published methodological research, iterative discussions, testing in systematic reviews, and feedback from the GRADE Working Group.
Approaches begin with a primary meta-analysis using a complete case analysis followed by sensitivity meta-analyses imputing, in each study, data for those with missing data, and then pooling across studies. For binary outcomes, we suggest use of “plausible worst case” in which review authors assume that those with missing data in treatment arms have proportionally higher event rates than those followed successfully. For continuous outcomes, imputed mean values come from other studies within the systematic review and the standard deviation (SD) from the median SDs of the control arms of all studies.
If the results of the primary meta-analysis are robust to the most extreme assumptions viewed as plausible, one does not rate down certainty in the evidence for risk of bias due to missing participant outcome data. If the results fail to prove robust to plausible assumptions, one would rate down certainty in the evidence for risk of bias.
A guidance was developed to identify participants with missing outcome data in randomized controlled trials
2019, Journal of Clinical Epidemiology
Citation Excerpt :
Impact of missing data on effect estimates in systematic reviews (unpublished data); Guidance for handling missing data of dichotomous [6] and continuous outcomes in systematic reviews [7,28]; Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance for assessing risk of bias associated with missing data in a body of evidence [8].
In order for authors of systematic reviews to address missing data in randomized controlled trials (RCTs), they need to first identify the number of trial participants with missing data. The objective of this study was to provide guidance for authors of systematic reviews on how to identify participants with missing outcome data in reports of RCTs.
Guidance statements were informed by a review of studies addressing the topic of missing data and an iterative process of feedback and refinement, through meetings involving experts in health research methodology and authors of systematic reviews.
The proposed guidance includes (1) definitions of key terms, (2) 19 categories of participants described in RCT reports and who might have missing data, and (3) a flowchart on how to judge the outcome data missingness for each category. The judgment of missingness relies on how trial authors report on the categories and handle them in their analyses. Practically, for their primary analysis, systematic review authors should choose how to identify participants with missing outcome data (i.e., use either “definitely missing data” or “total possible missing data”), then select a method for handling missing data in meta-analysis. Sensitivity analyses should be undertaken to explore consistency with competing options for classifying patients as having missing data.
Adopting the proposed guidance will help promote transparency and consistency regarding how missing data are managed in systematic reviews.
Efficacy of radiofrequency in lumbar facet joint pain: a systematic review and meta-analysis of placebo-controlled randomized controlled trials
2024, Radiologia Medica

View all citing articles on Scopus

: Conflict of interest: There is no financial or material support, commercial interest, or involvement with an organization with a financial interest in the research materials by any of the authors.

: Funding: There were no sources of funding for this study. S.E. is supported by an MITACS Elevate and a SickKids Restracomp award. P.A-.C. is funded by a Miguel Servet investigator contract from the Instituto de Salud Carlos III (CP09/00137).

View full text

Original ArticleAddressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers

Abstract

Background

Methods

Results

Conclusions

Introduction

Section snippets

Methods

Results

Discussion

Acknowledgments

J Clin Epidemiol

J Clin Epidemiol

Respir Med

Lancet

Lancet

Am J Med

Chest

Respir Med

Chest

Respir Med

Chest

LOST to follow-up Information in Trials (LOST–IT): potential impact on estimated treatment effects

BMJ

Further issues in meta-analysis: intention to treat issues. The Cochrane Collaboration open learning material

Addressing dichotomous data for participants excluded from trial analysis: a guide for systematic reviewers

PLoS One

Commentary–goodbye M(C)ID! Hello MID, where do you come from?

Health Serv Res

Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units

Health Qual Life Outcomes

Dealing with missing outcome data in randomized trials and observational studies

Am J Epidemiol

Imputation methods for missing outcome data in meta-analysis of clinical trials

Clin Trials

Chapter 8: risk of bias

CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials

BMJ

Original Article
Addressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers