Original Article
Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations

https://doi.org/10.1016/j.jclinepi.2015.05.029Get rights and content

Abstract

Objectives

Model specification—what adjusting variables are analytically modeled—may influence results of observational associations. We present a standardized approach to quantify the variability of results obtained with choices of adjustments called the “vibration of effects” (VoE).

Study Design and Setting

We estimated the VoE for 417 clinical, environmental, and physiological variables in association with all-cause mortality using National Health and Nutrition Examination Survey data. We selected 13 variables as adjustment covariates and computed 8,192 Cox models for each of 417 variables' associations with all-cause mortality.

Results

We present the VoE by assessing the variance of the effect size and in the −log10(P-value) obtained by different combinations of adjustments. We present whether there are multimodality patterns in effect sizes and P-values and the trajectory of results with increasing adjustments. For 31% of the 417 variables, we observed a Janus effect, with the effect being in opposite direction in the 99th versus the 1st percentile of analyses. For example, the vitamin E variant α-tocopherol had a VoE that indicated higher and lower risk for mortality.

Conclusion

Estimating VoE offers empirical estimates of associations are under different model specifications. When VoE is large, claims for observational associations should be very cautious.

Introduction

Observational associations between variables do not guarantee causality, and they are often complex and influenced by other variables (confounders and effect modifiers). Accounting for covariates is typically achieved through statistical modeling, such as multivariate regression. However, what variables should one choose to account for in complex multivariate phenomena where many variables may be confounded or correlated [1]? Model specification can be a major issue in diverse fields, including epidemiology [2], economics [3], [4], [5], and psychological science and neurosciences [6]. Thousands of associations are published, and many are often challenged and refuted by subsequent investigations [7], [8], [9]. Choices of models underlie our assumptions about association and about potential causes and effect [10]. Very often there is large uncertainty about what variables should be modeled and how they are related. Consequently, there is large heterogeneity in how investigators associate variables [2].

In discovery-based research in large data sets, there is often no prior evidence or biological plausibility on what adjustment variables to include in statistical models. In other cases, unequivocal evidence and plausibility may exist to include some adjustment variables in the model, lack of consensus on some others, and no available guidance on yet another set of adjustment variables. Interpretation of effects may vary depending on the analytical choices made. A way to compute the extent of instability of the results due to model specification is needed to guide inference.

The “vibration of effects” (VoE) [2] describes the extent to which an estimated association changes under multiple distinct analytical modeling approaches. The VoE is related also to the previously described concept of “multiple modeling” [9] or statistical model-induced variability [11]. To estimate the VoE empirically, we can compute the distribution of the point estimates of measures of association (e.g., relative risks, odds ratios) and P-values that are possible under different analytical scenarios. The VoE measures how susceptible an association is under different modeling scenarios; the larger the VoE, the greater the instability of the results. One may also explore which specific scenarios most influence the estimated association. Here, we describe a framework to systematically evaluate the VoE for a set of adjustment covariates.

Section snippets

Example of a controversial association

As an introductory example, we use the VoE framework to evaluate a contentious association between vitamin E (α-tocopherol) and mortality. Early publications of observational studies claimed large reductions in disease-related and mortality-related events in association with vitamin E [12], [13]. However, clinical trials that followed were not able to support the early observational findings [14], [15], [16], [17]. Furthermore, meta-analyses of clinical trials have showed nearly the opposite of

Data source: NHANES 1999–2000, 2001–2002, and 2003–2004

We downloaded National Health and Nutrition Examination Survey (NHANES) examination, laboratory, questionnaire, and National Death Index (NDI) linked mortality data for 1999–2000, 2001–2002, and 2003–2004 surveys. Mortality information was collected from the date of the survey participation through December 31, 2006, and ascertained via a probabilistic match between NHANES and NDI death certificate information. The NDI matches individuals on personal and demographic criteria, such as social

Estimating the VoE

VoE is estimated by computing the hazard ratio (HR) and P-value for a variable of interest while adjusting for all possible combinations of adjustments from a finite set of adjustment variables. Our algorithm for computing the VoE for a variable x (e.g., serum vitamin D) is shown in Fig. 1.

First, we downloaded 417 self-reported, clinical, and molecular measures with linked all-cause mortality information in participants from NHANES 1999–2004 (Fig. 1A). Mortality information was collected from

Discussion

Almost all reported findings in observational quantitative research to date in fields such as epidemiology consider only a single or a few modeling scenarios. It is often not clear whether this or these model(s) was/were selected a priori. It is often suspected that selective reporting abounds, that is, several models are tested and only those with the most impressive results are presented with particular attraction for nominally significant results [39]. There are ongoing efforts to enhance

Acknowledgments

The authors thank Profs. Andrew Gelman and Bin Yu for their comments. All data, software code, and additional figures can be found at the following website: http://chiragjpgroup.org/voe.

Author Contributions: C.J.P. and B.B. wrote the software code to conduct the VoE analysis. C.J.P., B.B., and J.P.A.I. came up with the idea and wrote/edited the manuscript.

References (58)

  • E. Leamer et al.

    Reporting the fragility of regression estimates

    Rev Econ Stat

    (1983)
  • H. Akil et al.

    Challenges and opportunities in mining neuroscience data

    Science

    (2011)
  • P. Boffetta et al.

    False-positive results in cancer epidemiology: a plea for epistemological modesty

    J Natl Cancer Inst

    (2008)
  • K. Fiedler

    Voodoo correlations are everywhere—not only in neuroscience

    Perspect Psychol Sci

    (2011)
  • S.S. Young et al.

    Deming, data and observational studies

    Significance

    (2011)
  • J. Pearl

    Causality: models, reasoning and inference

    (2009)
  • S.S. Young et al.

    Association of bisphenol A with diabetes and other abnormalities

    J Am Med Assoc

    (2009)
  • J.M. Gaziano

    Vitamin E and cardiovascular disease: observational studies

    Ann N Y Acad Sci

    (2004)
  • L. Sung et al.

    Vitamin E: the evidence for multiple roles in cancer

    Nutr Cancer

    (2003)
  • MRC/BHF Heart Protection Study of antioxidant vitamin supplementation in 20,536 high-risk individuals: a randomised placebo-controlled trial

    Lancet

    (2002)
  • S. Yusuf et al.

    Vitamin E supplementation and cardiovascular events in high-risk patients. The Heart Outcomes Prevention Evaluation Study Investigators

    N Engl J Med

    (2000)
  • J. Virtamo et al.

    Incidence of cancer and mortality following alpha-tocopherol and beta-carotene supplementation: a postintervention follow-up

    JAMA

    (2003)
  • E.R. Miller et al.

    Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality

    Ann Intern Med

    (2005)
  • G.G. Fillenbaum et al.

    Identifying a national death index match

    Am J Epidemiol

    (2009)
  • B.E. Ainsworth et al.

    Compendium of physical activities: an update of activity codes and MET intensities

    Med Sci Sports Exerc

    (2000)
  • C.J. Patel et al.

    Systematic evaluation of environmental factors: persistent pollutants and nutrients correlated with serum lipid levels

    Int J Epidemiol

    (2012)
  • 2008 Physical Activity Guidelines for Americans

    (2008)
  • U.S. Department of Agriculture, Agricultural Research Service, Beltsville Human Nutrition Research Center, Food Surveys...
  • U.S. Department of Agriculture, Agricultural Research Service, Beltsville Human Nutrition Research Center, Group FSR,...
  • Cited by (167)

    View all citing articles on Scopus

    Funding: This work was supported by a National Institute of Environmental Health Sciences grant K99 ES023504 and R21 ES0250252 and a PhRMA foundation award to C.J.P.

    Conflicts of interest: The authors declare no competing interests.

    View full text