Abstract
Variability in treatment effects is common in intervention studies using cluster randomized controlled trial (C-RCT) designs. Such variability is often examined in multilevel modeling (MLM) to understand how treatment effects (TRT) differ based on the level of a covariate (COV), called TRT \(\times \) COV. In detecting TRT \(\times \) COV effects using MLM, relationships between covariates and outcomes are assumed to vary across clusters linearly. However, this linearity assumption may not hold in all applications and an incorrect assumption may lead to biased statistical inference about TRT \(\times \) COV effects. In this study, we present generalized additive mixed model (GAMM) specifications in which cluster-specific functional relationships between covariates and outcomes can be modeled using by-variable smooth functions. In addition, the implementation for GAMM specifications is explained using the mgcv R package (Wood, 2021). The usefulness of the GAMM specifications is illustrated using intervention data from a C-RCT. Results of simulation studies showed that parameters and by-variable smooth functions were recovered well in various multilevel designs and the misspecification of the relationship between covariates and outcomes led to biased estimates of TRT \(\times \) COV effects. Furthermore, this study evaluated the extent to which the GAMM can be treated as an alternative model to MLM in the presence of a linear relationship.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We use a notation of a by-variable smooth function as f(x)factor following notations in GAMM (e.g., Wood, 2017, p. 326), which means that a smooth function \(f_{h}\) (where h is an index for a smooth function) is conditional on a factor.
Because a parametric random intercept is considered in Equation 2, the unconditional GAMM is the same as the unconditional MLM.
We use the abbreviation RMSEI to distinguish it from the root mean squared error of an estimator (RMSE) in the simulation study.
In this study, we did not consider relative bias due to its scaling artifact. When true parameters are close to 0 (e.g., \(\tau _{00}=\)0.032, 0.067, and 0.257 and basis coefficients for smooth functions in the simulation study), relative bias will be inflated by the true parameters close to 0 in the denominator even for small differences from the true parameters.
Exceptional cases include bias of \(\widehat{\gamma }_{00}\) and \(\widehat{\gamma }_{02}\) with respect to \(n_{j}\) and \(\widehat{\sigma }^{2}\) with respect to J in MLM; and bias of \(\widehat{\gamma }_{00}\) with respect to J and \(n_{j}\), \(\widehat{\gamma }_{02}\) with respect to J, and \(\widehat{\sigma }^{2}\) with respect to \(n_{j}\) in GAMM.
For a single replication of GAMM estimation under GAMM as a data-generating model in a simulation condition with \(J=80\), about an hour was required on a laptop computer with a 2.8 GHz Intel Core i7 CPU and 16 GB of RAM.
References
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723. https://doi.org/10.1109/TAC.1974.1100705
Baranov, V., Bhalotra, S., Biroli, P., & Maselko, J. (2020). Maternal depression, women’s empowerment, and parental investment: Evidence from a randomized controlled trial. American Economic Review, 110(3), 824–859. https://doi.org/10.1257/aer.20180511
Baranov, V., Bhalotra, S., Biroli, P., & Maselko, J. (2020b). Data and code for: Maternal depression, women’s empowerment, and parental investment: Evidence from a randomized controlled trial. Nashville, TN: American Economic Association [publisher], 2020. Ann Arbor, MI: Inter-University Consortium for Political and Social Research [distributor], 2020-02-28. https://doi.org/10.3886/E111366V1
Bloom, H. S., & Spybrook, J. (2017). Assessing the precision of multisite trials for estimating the parameters of a cross-site population distribution of program effects. Journal of Research on Educational Effectiveness, 10(4), 877–902. https://doi.org/10.1080/19345747.2016.1271069
Cho, S.-J., Preacher, K. J., Yaremych, H. E., Naveiras, M., Fuchs, D., & Fuchs, L. S. (2022). Modeling multilevel nonlinear treatment-by-covariate interactions in cluster randomized controlled trials using a generalized additive mixed model. British Journal of Mathematical and Statistical Psychology. https://doi.org/10.1111/bmsp.12265
Fine, K. L., Suk, H. W., & Grimm, K. J. (2019). An examination of a functional mixed-effects modeling approach to the analysis of longitudinal data. Multivariate Behavioral Research, 54(4), 475–491. https://doi.org/10.1080/00273171.2018.1520626
Fuchs, D., Cho, E., Toste, J. R., Fuchs, L. S., Gilbert, J. K., McMaster, K. L., Svenson, E., & Thompson, A. (2021). A quasiexperimental evaluation of two versions of first-grade PALS: One with and one without repeated reading. Exceptional Children, 87(2), 141–162. https://doi.org/10.1177/0014402920921828
Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
Guo, W. (2002). Functional mixed effects models. Biometrics, 58, 121–128. https://doi.org/10.1111/j.0006-341X.2002.00121.x
Harrell, F. E. (2015). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (Vol. 3). Springer. https://doi.org/10.1007/978-3-319-19425-7
Kimeldorf, G. S., & Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. The Annals of Mathematical Statistics, 41(2), 495–502. https://doi.org/10.1214/aoms/1177697089
Lawrence, J. F., Francis, D., Paré-Blagoev, J., & Snow, C. E. (2017). The poor get richer: Heterogeneity in the efficacy of a school-level intervention for academic language. Journal of Research on Educational Effectiveness, 10(4), 767–793. https://doi.org/10.1080/19345747.2016.1237596
Lin, X., & Zhang, D. (1999). Inference in generalized additive mixed models by using smoothing splines. Journal of the Royal Statistical Society, 61(2), 381–400. https://doi.org/10.1111/1467-9868.00183
Longford, N. T. (1993). Random coefficient models. Oxford University Press.
Pinheiro, J., Bates, D., DebRoy, S., & Sarkar, D. (2021). nlme: Linear and nonlinear mixed effects models. R package version 3. 1–152. https://CRAN.R-project.org/package=nlme
Preacher, K. J., & Sterba, S. K. (2019). Aptitude-by-treatment interactions in research on educational interventions. Exceptional Children, 85(2), 248–264. https://doi.org/10.1177/0014402918802803
R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
Ramsay, J. O., & Silverman, B. W. (2002). Applied functional data analysis: Methods and case studies. Springer. https://doi.org/10.1007/b98886
Ramsay, J. O., & Silverman, B. W. (2005). Functional data analysis (2nd ed.). Springer. https://doi.org/10.1007/b98888
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Sage.
Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199–213. https://doi.org/10.1037/1082-989X.5.2.199
Silverman, B. W. (1985). Some aspects of the spline smoothing approach to nonparametric regression curve fitting. Journal of the Royal Statistical Society, Series B, 47(1), 1–21. https://doi.org/10.1111/j.2517-6161.1985.tb01327.x
Spybrook, J., Kelcey, B., & Dong, N. (2016). Power for detecting treatment by moderator effects in two-and three-level cluster randomized trials. Journal of Educational and Behavioral Statistics, 41(6), 605–627. https://doi.org/10.3102/1076998616655442
Tipton, E., & Hedges, L. V. (2017). The role of the sample in estimating and explaining treatment effect heterogeneity. Journal of Research on Educational Effectiveness, 10(4), 903–906. https://doi.org/10.1080/19345747.2017.1364563
Wahba, G. (1983). Bayesian “confidence intervals” for the cross-validated smoothing spline. Journal of the Royal Statistical Society: Series B (Methodological), 45(1), 133–150. https://doi.org/10.1111/j.2517-6161.1983.tb01239.x
Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719
Wood, S. N. (2013). A simple test for random effects in regression models. Biometrika, 100, 1005–1010. https://doi.org/10.1093/biomet/ast038
Wood, S. N. (2017). Generalized additive models: An introduction with R (2nd ed.). Chapman & Hall/CRC. https://doi.org/10.1201/9781315370279
Wood, S. N. (2021). Package ‘mgcv.’ Retrieved from https://cran.r-project.org/web/packages/mgcv/mgcv.pdf
Wood, S. N., Pya, N., & Säfken, B. (2016). Smoothing parameter and model selection for general smooth models. Journal of the American Statistical Association, 111(516), 1548–1563. https://doi.org/10.1080/01621459.2016.1180986
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cho, SJ., Preacher, K.J., Yaremych, H.E. et al. Modeling variability in treatment effects for cluster randomized controlled trials using by-variable smooth functions in a generalized additive mixed model. Behav Res 56, 2094–2113 (2024). https://doi.org/10.3758/s13428-023-02138-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3758/s13428-023-02138-w