Skip to main content
SpringerLink
Log in

Comparing DIC and WAIC for multilevel models with missing data

  • Original Manuscript
  • Published:
Behavior Research Methods Aims and scope Submit manuscript

Abstract

In Bayesian statistics, the most widely used criteria of Bayesian model assessment and comparison are Deviance Information Criterion (DIC) and Watanabe–Akaike Information Criterion (WAIC). We use a multilevel mediation model as an illustrative example to compare different types of DIC and WAIC. More specifically, we aim to compare the performance of conditional and marginal DICs and WAICs, and investigate their performance with missing data. We focus on two versions of DIC (\(DIC_{1}\) and \(DIC_{2}\)) and one version of WAIC. In addition, we explore whether it is necessary to include the nuisance models of incomplete exogenous variables in likelihood. Based on the simulation results, whether \(DIC_{2}\) is better than \(DIC_{1}\) and WAIC and whether we should include the nuisance models of exogenous variables in likelihood functions depend on whether we use marginal or conditional likelihoods. Overall, we find that the marginal likelihood based-\(DIC_{2}\) that excludes the likelihood of covariate models generally had the highest true model selection rates.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. lavaan does not allow treating predictors/covariates as random when the model contains nonlinear covariate effects.

References

  • Asparouhov, T., & Muthén, B. (2010). Bayesian analysis using mplus: Technical implementation (version 3). Citeseer.

  • Bartlett, J. W., Seaman, S. R., White, I. R., & Carpenter, J. R. (2015). Multiple imputation of covariates by fully conditional specification: accommodating the substantive model. Statistical Methods in Medical Research, 24(4), 462–487.

    Article  PubMed  PubMed Central  Google Scholar 

  • Cain, M. K., & Zhang, Z. (2019). Fit for a bayesian: An evaluation of ppp and dic for structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 26(1), 39–50.

    Article  Google Scholar 

  • Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1), 1–32.

    Article  PubMed  PubMed Central  Google Scholar 

  • Celeux, G., Forbes, F., Robert, C. P., & Titterington, D. M. (2006). Deviance information criteria for missing data models. Bayesian Analysis, 1(4), 651–673.

    Article  Google Scholar 

  • Du, H., Alacam, E., Mena, S., & Keller, B. T. (2022). Compatibility in imputation specification. Behavior Research Methods, 1–19.

  • Enders, C. K. (2022). Applied missing data analysis (2nd ed.). New York: Guilford press.

    Google Scholar 

  • Enders, C. K., Du, H., & Keller, B. T. (2020). A model-based imputation procedure for multilevel regression models with random coefficients, interaction effects, and non-linear terms. Psychological Methods, 25(1), 88–112.

    Article  PubMed  Google Scholar 

  • Enders, C. K., Hayes, T., & Du, H. (2018). A comparison of multilevel imputation schemes for random coefficient models: Fully conditional specification and joint model imputation with random covariance matrices. Multivariate Behavioral Research, 53(5), 695–713.

    Article  PubMed  Google Scholar 

  • Enders, C. K., Keller, B. T., & Woller, M. T. (2023). A simple “monte carlo” method for estimating power in multilevel designs. Revised manuscript submitted for publication to Psychological methods.

  • Erler, N. S., Rizopoulos, D., Jaddoe, V. W., Franco, O. H., & Lesaffre, E. M. (2019). Bayesian imputation of time-varying covariates in linear mixed models. Statistical Methods in Medical Research, 28(2), 555–568.

    Article  PubMed  Google Scholar 

  • Erler, N. S., Rizopoulos, D., Rosmalen, Jv., Jaddoe, V. W., Franco, O. H., & Lesaffre, E. M. (2016). Dealing with missing covariates in epidemiologic studies: a comparison between multiple imputation and a full bayesian approach. Statistics in Medicine, 35(17), 2955–2974.

    Article  PubMed  Google Scholar 

  • Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2014). Bayesian data analysis (vol. 2). London: Chapman & Hall.

  • Gelman, A., Hwang, J., & Vehtari, A. (2014). Understanding predictive information criteria for bayesian models. Statistics and Computing, 24(6), 997–1016.

    Article  Google Scholar 

  • Goldstein, H., Carpenter, J. R., & Browne, W. J. (2014). Fitting multilevel multivariate models with missing data in responses and covariates that may include interactions and non-linear terms. Journal of the Royal Statistical Society: Series A (Statistics in Society), 177(2), 553–564.

    Article  Google Scholar 

  • Gronau, Q. F., & Wagenmakers, E.-J. (2019). Limitations of bayesian leave-one-out cross-validation for model selection. Computational Brain & Behavior, 2, 1–11.

    Article  Google Scholar 

  • Grund, S., Lüdtke, O., & Robitzsch, A. (2018). Multiple imputation of missing data for multilevel models: Simulations and recommendations. Organizational Research Methods, 21(1), 111–149.

    Article  Google Scholar 

  • Grund, S., Lüdtke, O., & Robitzsch, A. (2021). Multiple imputation of missing data in multilevel models with the r package mdmb: a flexible sequential modeling approach. Behavior Research Methods, 53, 2631–2649.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ibrahim, J. G., Chen, M.-H., & Lipsitz, S. R. (1999). Monte carlo em for missing covariates in parametric regression models. Biometrics, 55(2), 591–596.

    Article  PubMed  Google Scholar 

  • Keller, B. T., & Enders, C. K. (2021). Blimp user’s guide (version 3). Retrieved from www.appliedmissingdata.com/multilevel-imputation.html. Accessed 01 Sept 2023

  • Kim, S., Belin, T. R., & Sugar, C. A. (2018). Multiple imputation with non-additively related variables: Joint-modeling and approximations. Statistical Methods in Medical Research, 27(6), 1683–1694.

    Article  PubMed  Google Scholar 

  • Kim, S., Sugar, C. A., & Belin, T. R. (2015). Evaluating model-based imputation methods for missing covariates in regression models with interactions. Statistics in Medicine, 34(11), 1876–1888.

    Article  PubMed  PubMed Central  Google Scholar 

  • Li, L., Qiu, S., Zhang, B., & Feng, C. X. (2016). Approximating cross-validatory predictive evaluation in bayesian latent variable models with integrated is and waic. Statistics and Computing, 26(4), 881–897.

    Article  Google Scholar 

  • Lu, L., & Zhang, Z. (2022). How to select the best fit model among bayesian latent growth models for complex data. Journal of Behavioral Data Science, 2(1), 35–58.

    Article  Google Scholar 

  • Lüdtke, O., Robitzsch, A., & West, S. G. (2020). Regression models involving nonlinear effects with missing data: A sequential modeling approach using bayesian estimation. Psychological Methods, 25(2), 157–181.

    Article  PubMed  Google Scholar 

  • Merkle, E. C., Furr, D., & Rabe-Hesketh, S. (2019). Bayesian comparison of latent variable models: Conditional versus marginal likelihoods. Psychometrika, 84(3), 802–829.

    Article  PubMed  Google Scholar 

  • Merkle, E. C., & Rosseel, Y. (2015). blavaan: Bayesian structural equation models via parameter expansion. arXiv:1511.05604.

  • Millar, R. B. (2018). Conditional vs marginal estimation of the predictive loss of hierarchical models using waic and cross-validation. Statistics and Computing, 28(2), 375–385.

    Article  Google Scholar 

  • Muthén, L., & Muthén, B. (2017). Mplus userâs guide (8th ed.). Los Angeles, CA: Author.

    Google Scholar 

  • Plummer, M., et al. (2003). Jags: A program for analysis of bayesian graphical models using gibbs sampling. In Proceedings of the 3rd international workshop on distributed statistical computing (vol. 124, pp. 1–10).

  • Rights, J. D., & Sterba, S. K. (2019). Quantifying explained variance in multilevel models: An integrative framework for defining r-squared measures. Psychological methods, 24(3), 309.

  • Seaman, S. R., Bartlett, J. W., & White, I. R. (2012). Multiple imputation of missing covariates with non-linear effects and interactions: an evaluation of statistical methods. BMC Medical Research Methodology, 12(1), 46.

  • Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Linde, A. (2014). The deviance information criterion: 12 years on. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(3), 485–493.

    Article  Google Scholar 

  • Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583–639.

  • Spiegelhalter, D. J., Thomas, A., Best, N. G., Gilks, W., & Lunn, D. (1996). Bugs: Bayesian inference using gibbs sampling. Version 0.5,(version ii) http://www.mrc-bsu.cam.ac.uk/bugs,19.

  • Tong, X., Kim, S., & Ke, Z. (2022). Impact of likelihoods on class enumeration in bayesian growth mixture modeling. In M. Wiberg, D. Molenaar, J. González, J.-S. Kim, & H. Hwang (Eds.), Quantitative psychology (pp. 111–120). Cham: Springer International Publishing.

    Chapter  Google Scholar 

  • Van Buuren, S., Brand, J. P., Groothuis-Oudshoorn, C. G., & Rubin, D. B. (2006). Fully conditional specification in multivariate imputation. Journal of Statistical Computation and Simulation, 76(12), 1049–1064.

    Article  Google Scholar 

  • Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical bayesian model evaluation using leave-one-out cross-validation and waic. Statistics and Computing, 27, 1413–1432.

    Article  Google Scholar 

  • Vehtari, A., Mononen, T., Tolvanen, V., Sivula, T., & Winther, O. (2016). Bayesian leave-one-out cross-validation approximations for gaussian latent variable models. The Journal of Machine Learning Research, 17(1), 3581–3618.

    Google Scholar 

  • Vehtari, A., Gabry, J., Magnusson, M., Yao, Y., BÃOErkner, P.-C., Paananen, T., ... Gelman, A. (2023). loo: Efficient leave-one-out cross-validation and waic for bayesian models. Retrieved from https://mc-stan.org/loo/ (R package version 2.6.0)

  • Watanabe, S., & Opper, M. (2010). Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11(12).

  • Zhang, X., Tao, J., Wang, C., & Shi, N.-Z. (2019). Bayesian model selection methods for multilevel irt models: A comparison of five dic-based indices. Journal of Educational Measurement, 56(1), 3–27.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Institute of Educational Sciences award R305D190002.

Author information

Authors and Affiliations

  1. Department of Psychology, UCLA, Los Angeles, CA, 90095, USA

    Han Du, Egamaria Alacam & Craig Enders

  2. Department of Educational, School, & Counseling Psychology, University of Missouri, Columbia, Missouri, 65201, USA

    Brian Keller

Authors
  1. Han Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian Keller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Egamaria Alacam
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Craig Enders
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Han Du.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Du, H., Keller, B., Alacam, E. et al. Comparing DIC and WAIC for multilevel models with missing data. Behav Res (2023). https://doi.org/10.3758/s13428-023-02231-0

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.3758/s13428-023-02231-0

Keywords

Search

Navigation