Elsevier

Journal of Clinical Epidemiology

Volume 79, November 2016, Pages 41-45
Journal of Clinical Epidemiology

Original Article
Evidence of nicotine replacement's effectiveness dissolves when meta-regression accommodates multiple sources of bias

https://doi.org/10.1016/j.jclinepi.2016.03.024Get rights and content

Abstract

Objectives

To accommodate and correct identifiable bias and risks of bias among clinical trials of nicotine replacement therapy (NRT).

Study Design and Setting

Meta-regression analysis of a published Cochrane Collaboration systematic review of 122 placebo-controlled clinical trials.

Results

Both identified risks of bias and potential publication (or reporting or small sample) bias are associated with an increase in the reported effectiveness of NRT. Whenever multiple sources of biases are accommodated by meta-regression, no evidence of a practically notable or statistically significant overall increased rate of smoking cessation remains. Our findings are in stark contrast with the 50% to 70% increase in smoking cessation reported by the Cochrane Collaboration systematic review.

Conclusion

After more than 100 randomized clinical trials have been conducted, the overall effectiveness of NRT is in doubt. Simple, well-established meta-regression methods can test, accommodate, and correct multiple sources biases, often mentioned but dismissed by conventional systematic reviews.

Introduction

Smoking tobacco is the leading cause of preventable death in the United States [1]. Yet, quitting smoking is difficult for those addicted to nicotine. A number of nicotine replacement therapies (NRTs) are available to help smokers quit, which the World Health Organization regards as essential medicine [2]. In general, NRTs are considered effective. The most recent and authoritative systematic review concludes that NRT increases “the rate of long-term quitting by approximately 50% to 70% regardless of setting” ([3], p. 23, Authors' Conclusions). Nonetheless, when meta-regression is used on all the NRT vs. placebo comparisons from this same Cochrane review, little evidence remains that NRT increases smoking cessation. Meta-regression analysis (MRA) can go beyond state-of-the-art systematic reviews by simultaneously accommodating both risks of bias and small sample, reporting, or publication bias.

Risks of bias refer to the routine assessment of potential limitations or weaknesses in how clinical trials are conducted [4]. Cochrane Collaboration systematic reviews are expected to code for these potential threats to the validity of randomized clinical trials (RCTs). Publication bias concerns the selective reporting of statistically significant findings [5], [6], [7], [8], [9], [10]. It represents a different source of potential bias in RCTs, one, that is, expected to operate in a “positive” direction. Cochrane Collaboration systematic reviewers are also asked to assess the threat of publication bias.

Is NRT better than a placebo? Are there differences among the types of NRT? Are clinical trials selectively reported or published to show that NRT has statistically positive effects on smoking cessation? We find that when potential biases from multiple sources are simultaneously accounted for, statistical traces of NRTs effectiveness dissolve.

This study statistically analyzes 122 NRT trial results published in a Cochrane review ([3], Figure 2, p. 14). One hundred twenty of these findings come from NRT vs. placebo comparisons. The remaining two compare a combination of NRT to no NRT, “which did not affect the overall estimate” ([3], Figure 2, p. 13). We use all 122 NRT effect sizes because they form the basis of the conclusion by Stead et al. [3], and we wish to introduce no selective reporting bias. In addition to risk ratios and their confidence intervals, Stead et al. [3] classify the risk of bias for each clinical trial of NRT and report the type of NRT used: patch, gum, nasal spray, lozenge, oral spray, and inhaler. We re-evaluate this Cochrane review using a meta-regression model that is capable of simultaneously filtering out potential biases and risks of bias. When multiple vectors for bias are explicitly allowed, clear evidence of the overall effectiveness of NRT disappears.

Section snippets

Meta-regression

MRA is used to allow multiple dimensions of NRT research to be considered simultaneously. Meta-regression allows us to accommodate the effects of: publication bias, reporting bias, small-sample bias, identified risks of bias, and heterogeneity, simultaneously, on NRT effectiveness. We also corroborate our multiple meta-regression findings by investigating subsets of high-quality research.

The simple Egger meta-regression has often been used to detect publication (or small sample or reporting)

Results

First, we combine only those studies regarded as having low risk of bias, as judged by Stead et al. [3]. The largest subset of these is Q2, containing 46 RCTs. Among these 46 log RRs, the fixed-effect and random-effects weighted averages imply that smoking cessation will increase by 51% and 54%, respectively, with 95% confidence intervals (CIs) = [(36% to 68%); (36% to 73%)]. Although somewhat smaller, both weighted averages' CIs overlap with conclusions by Stead et al. [3] and represent a

Discussion and conclusion

We re-analyze a comprehensive and extensive systematic Cochrane review of NRT but come to a very different conclusion. Rather than finding NRT to be effective, increasing “the rate of long-term quitting by approximately 50% to 70% regardless of setting” ([3], p. 23), we find no overall evidence of increased smoking cessation after potential biases and risks of bias are fully considered.

A critic might point out that the Egger test for publication or small-sample bias is known to have low power.

Acknowledgments

T.D. Stanley acknowledges support from the Czech Science Foundation (grant 15-02411S).

References (16)

  • S. Redmond et al.

    Cohort study of trials submitted to ethics committee identified discrepant reporting of outcomes in publications

    J Clin Epidemiol

    (2013)
  • A.H. Mokdad et al.

    Actual causes of death in the United States, 2000

    JAMA

    (2004)
  • WHO model list of essential medicines, 18th list

    (2013)
  • L.F. Stead et al.

    Nicotine replacement therapy for smoking cessation

    Cochrane Database Syst Rev

    (2012)
  • J.P. Higgins et al.

    The Cochrane Collaboration's tool for assessing risk of bias in randomised trials

    BMJ

    (2011)
  • H. Doucouliagos et al.

    Publication selection bias in minimum-wage research? A meta-regression analysis

    Brit J Ind Relat

    (2009)
  • M. Egger et al.

    Bias in meta-analysis detected by a simple, graphical test

    BMJ

    (1997)
  • T. Havranek

    Measuring intertemporal substitution: the importance of method choices and selective reporting

    J Eur Econ Assoc

    (2015)
There are more references available in the full text version of this article.

Cited by (0)

View full text