Abstract
Given the need to develop and validate effective implementation models that lead to sustainable improvements, we prospectively examined changes in attitudes, behaviors, and perceived organizational support during and after statewide Community-Based Learning Collaboratives (CBLCs) promoting trauma-focused evidence-based practices (EBPs). Participants (N = 857; i.e., 492 clinicians, 218 brokers, and 139 senior leaders) from 10 CBLCs completed surveys pre- and post-CBLC; a subsample (n = 146) completed a follow-up survey approximately two years post-CBLC. Results indicated (a) medium, sustained increases in clinician-reported use of trauma-focused EBPs, (b) medium to large, sustained increases in perceived organizational support for trauma-focused EBPs, and (c) trivial to small, sustained increases in perceived organizational support for EBPs broadly. In contrast, clinician-reported overall attitudes towards EBPs decreased to a trivial degree pre- to post-CBLC, but then increased to a small, statistically significant degree from post-CBLC to follow-up. Notably, the degree of perceived improvements in organizational support for general and trauma-focused EBPs varied by professional role. Findings suggest the CBLC implementation strategies may both increase and sustain provider practices and organizational support towards EBPs, particularly those EBPs a CBLC explicitly targets.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aarons, G. A. (2005). Measuring provider attitudes toward evidence-based practice: Consideration of organizational context and individual differences. Child and Adolescent Psychiatric Clinics of North America,14(2), 255–271. https://doi.org/10.1016/j.chc.2004.04.008.
Aarons, G. A., Glisson, C., Hoagwood, K. E., Kelleher, K., Landsverk, J., & Cafri, G. (2010). Psychometric properties and US national norms of the evidence-based practice attitude scale (EBPAS). Psychological Assessment,22(2), 356–365. https://doi.org/10.1037/a0019188.
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health,38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7.
Aarons, G. A., & Sawitzky, A. C. (2006). Organizational climate partially mediates the effect of culture on work attitudes and staff turnover in mental health services. Administration and Policy in Mental Health,33(3), 289–301. https://doi.org/10.1007/s10488-006-0039-1.
Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology,77(2), 270–280. https://doi.org/10.1037/a0013223.
Amaya-Jackson, L., Hagele, D., Sideris, J., Potter, D., Briggs, E. C., Keen, L., et al. (2018). Pilot to policy: Statewide dissemination and implementation of evidence-based treatment for traumatized youth. BMC Health Services Research,18(1), 589. https://doi.org/10.1186/s12913-018-3395-0.
Beidas, R. S., Williams, N. J., Green, P. D., Aarons, G. A., Becker-Haimes, E. M., Evans, A. C., et al. (2018). Concordance between administrator and clinician ratings of organizational culture and climate. Administration and Policy in Mental Health,45(1), 142–151. https://doi.org/10.1007/s10488-016-0776-8.
Brown, C. H., Chamberlain, P., Saldana, L., Padgett, C., Wang, W., & Cruden, G. (2014). Evaluation of two implementation strategies in 51 child county public service systems in two states: Results of a cluster randomized head-to-head implementation trial. Implementation Science,9(1), 134. https://doi.org/10.1186/s13012-014-0134-8.
Centers for Disease Control and Prevention. (2013). Mental health surveillance among children—United States, 2005–2011. MMWR Morbidity and Mortality Weekly Report,62(Suppl 2), 1–35.
Chamberlain, P., Roberts, R., Jones, H., Marsenich, L., Sosna, T., & Price, J. M. (2012). Three collaborative models for scaling up evidence-based practices. Administration and Policy in Mental Health,39(4), 278–290. https://doi.org/10.1007/s10488-011-0349-9.
Cohen, J. A., Mannarino, A. P., & Deblinger, E. (2017). Treating trauma and traumatic grief in children and adolescents (2nd ed.). New York: Guilford Press.
Daily, S., Tout, K., Douglass, A., Miranda, B., Halle, T., Agosti, J., et al. (2018). Culture of Continuous Learning Project: A Literature Review of the Breakthrough Series Collaborative (BSC), OPRE Report #2018-28. Washington, DC. Retrieved from https://www.acf.hhs.gov/opre.
Deblinger, E., Cohen, J. A., Runyon, M., & Hanson, R. F. (2005). Clinical Practices Questionnaire. Unpublished instrument.
Dong, Y., & Peng, C.-Y. J. (2013). Principled missing data methods for researchers. SpringerPlus, 2(1), 222. https://doi.org/10.1186/2193-1801-2-222.
Ebert, L., Amaya-Jackson, L., Markiewicz, J. M., Kisiel, C., & Fairbank, J. A. (2012). Use of the breakthrough series collaborative to support broad and sustained use of evidence-based trauma treatment for children in community practice settings. Administration and Policy in Mental Health,39(3), 187–199. https://doi.org/10.1007/s10488-011-0347-y.
Fergusson, D. M., Boden, J. M., & Horwood, L. J. (2008). Exposure to childhood sexual and physical abuse and adjustment in early adulthood. Child Abuse & Neglect,32(6), 607–619. https://doi.org/10.1016/j.chiabu.2006.12.018.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida.
Glisson, C., & Green, P. D. (2011). Organizational climate, services, and outcomes in child welfare systems. Child Abuse & Neglect,35(8), 582–591. https://doi.org/10.1016/j.chiabu.2011.04.009.
Glisson, C., Schoenwald, S. K., Hemmelgarn, A. L., Green, P. D., Dukes, D., Armstrong, K. S., et al. (2010). Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology,78(4), 537–550. https://doi.org/10.1037/a0019160.
Glisson, C., Schoenwald, S. K., Kelleher, K., Landsverk, J., Hoagwood, K. E., Mayberg, S., et al. (2008). Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Administration and Policy in Mental Health,35(1–2), 124–133. https://doi.org/10.1007/s10488-007-0152-9.
Hammick, M., Freeth, D., Koppel, I., Reeves, S., & Barr, H. (2007). A best evidence systematic review of interprofessional education: BEME Guide no. 9. Medical Teacher,29(8), 735–751. https://doi.org/10.1080/01421590701682576.
Hanson, R. F., Saunders, B. E., Peer, S. O., Ralston, E., Moreland, A. D., Schoenwald, S., & Chapman, J. (2018). Community-based learning collaboratives and participant reports of interprofessional collaboration, barriers to, and utilization of child trauma services. Children and Youth Services Review, 94, 306–314. https://doi.org/10.1016/j.childyouth.2018.09.038.
Hanson, R. F., Saunders, B. E., Ralston, E., Moreland, A. D., Peer, S. O., & Fitzgerald, M. M. (2019). Statewide implementation of child trauma-focused practices using the community-based learning collaborative model. Psychological Services, 16(1), 170–181. https://doi.org/10.1037/ser0000319.
Hanson, R. F., Schoenwald, S. K., Saunders, B. E., Chapman, J., Palinkas, L. A., Moreland, A. D., & Dopp, A. R. (2016). Testing the Community-Based Learning Collaborative (CBLC) implementation model: A study protocol. International Journal of Mental Health Systems, 10(1), 52. https://doi.org/10.1186/s13033-016-0084-4.
Herschell, A. D., Kolko, D. J., Scudder, A. T., Taber-Thomas, S., Schaffner, K. F., Hiegel, S. A., et al. (2015). Protocol for a statewide randomized controlled trial to compare three training models for implementing an evidence-based treatment. Implementation Science,10(1), 133. https://doi.org/10.1186/s13012-015-0324-z.
Hoagwood, K. E., Olin, S. S., Horwitz, S., McKay, M., Cleek, A., Gleacher, A., et al. (2014). Scaling up evidence-based practices for children and families in New York State: Toward evidence-based policies on implementation for state mental health systems. Journal of Clinical Child and Adolescent Psychology,43(2), 145–157. https://doi.org/10.1080/15374416.2013.869749.
Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed., Vol. 31). New York, NY: Routledge.
Hurlburt, M. S., Leslie, L. K., Landsverk, J., Barth, R. P., Burns, B. J., Gibbons, R. D., et al. (2004). Contextual predictors of mental health service use among children open to child welfare. Archives of General Psychiatry,61(12), 1217–1224. https://doi.org/10.1001/archpsyc.61.12.1217.
IBM Corp. (2017). IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp.
Institute for Healthcare Improvement. (2003). The Breakthrough Series: IHI’s collaborative model for achieving breakthrough improvement. Boston, MA: Institute for Healthcare Improvement.
Jaramillo, J. Á., Sossa, J. W. Z., & Mendoza, G. L. O. (2019). Barriers to sustainability for small and medium enterprises in the framework of sustainable development—Literature review. Business Strategy and the Environment,28(4), 512–524. https://doi.org/10.1002/bse.2261.
Lang, J. M., Campbell, K., Shanley, P., Crusto, C. A., & Connell, C. M. (2016). Building capacity for trauma-informed care in the child welfare system: Initial results of a statewide implementation. Child Maltreatment,21(2), 113–124. https://doi.org/10.1177/1077559516635273.
Lang, J. M., Franks, R. P., Epstein, C., Stover, C., & Oliver, J. A. (2015). Statewide dissemination of an evidence-based practice using Breakthrough Series Collaboratives. Children and Youth Services Review,55, 201–209. https://doi.org/10.1016/j.childyouth.2015.06.005.
Lang, J. M., Randall, K. G., Delaney, M., & Vanderploeg, J. J. (2017). A model for sustaining evidence-based practices in a statewide system. Families in Society,98(1), 18–26. https://doi.org/10.1606/1044-3894.2017.5.
Markiewicz, J. M., Ebert, L., Ling, D., Amaya-Jackson, L., & Kisiel, C. (2006). Learning Collaborative Toolkit. Los Angeles, CA: In National Center for Child Traumatic Stress.
Merikangas, K. R., He, J.-P., Brody, D., Fisher, P. W., Bourdon, K., & Koretz, D. S. (2010). Prevalence and treatment of mental disorders among US children in the 2001–2004 NHANES. Pediatrics,125(1), 75–81. https://doi.org/10.1542/peds.2008-2598.
Nadeem, E., Olin, S. S., Hill, L. C., Hoagwood, K. E., & Horwitz, S. M. (2013). Understanding the components of quality improvement collaboratives: A systematic literature review. The Milbank Quarterly,91(2), 354–394. https://doi.org/10.1111/milq.12016.
Nadeem, E., Olin, S. S., Hill, L. C., Hoagwood, K. E., & Horwitz, S. M. (2014). A literature review of learning collaboratives in mental health care: Used but untested. Psychiatric Services (Washington, DC),65(9), 1088–1099. https://doi.org/10.1176/appi.ps.201300229.
Proctor, E. K., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G. A., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health,38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.
Saunders, B. E., & Adams, Z. W. (2014). Epidemiology of traumatic experiences in childhood. Child and Adolescent Psychiatric Clinics of North America,23(2), 167–184. https://doi.org/10.1016/j.chc.2013.12.003.
Saunders, B. E., & Hanson, R. F. (2014). Innovative methods for implementing evidence-supported interventions for mental health treatment of child and adolescent victims of violence. In R. M. Reece, R. F. Hanson, & J. Sargent (Eds.), Treatment of child abuse: Common ground for mental health, medical, and legal practitioners (2nd ed., pp. 235–245). Baltimore, MD: Johns Hopkins University Press.
Scheirer, M. A., & Dearing, J. W. (2011). An agenda for research on the sustainability of public health programs. American Journal of Public Health,101(11), 2059–2067. https://doi.org/10.2105/AJPH.2011.300193.
Shelton, R. C., Cooper, B. R., & Wiltsey-Stirman, S. (2018). The sustainability of evidence-based interventions and practices in public health and health care. Annual Review of Public Health,39(1), 55–76. https://doi.org/10.1146/annurev-publhealth-040617-014731.
Steiger, J. H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods,9(2), 164–182. https://doi.org/10.1037/1082-989X.9.2.164.
Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2010). The sustainability of evidence-based practices in routine mental health agencies. Community Mental Health Journal,46(2), 119–129. https://doi.org/10.1007/s10597-009-9202-y.
West, S. G., Cham, H., & Liu, Y. (2014). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (2nd ed., pp. 49–80). New York, NY: Cambridge University Press.
Wiltsey-Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science,7(1), 17. https://doi.org/10.1186/1748-5908-7-17.
Woltmann, E. M., Whitley, R., McHugo, G. J., Brunette, M., Torrey, W. C., Coots, L., et al. (2008). The role of staff turnover in the implementation of evidence-based practices in mental health care. Psychiatric Services,59(7), 732–737. https://doi.org/10.1176/ps.2008.59.7.732.
Wuensch, K. L. (2012). Using SPSS to obtain a confidence interval for Cohen’s d. Retrieved from http://core.ecu.edu/psyc/wuenschk/SPSS/CI-d-SPSS.pdf.
Zwarenstein, M., Goldman, J., & Reeves, S. (2009). Interprofessional collaboration: Effects of practice-based interventions on professional practice and healthcare outcomes. In M. Zwarenstein (Ed.), Cochrane Database of Systematic Reviews (Vol. 3). Chichester, UK: Wiley. https://doi.org/10.1002/14651858.CD000072.pub2
Acknowledgements
This research was supported in part by grants from the Duke Endowment (1790-SP) and the National Institute of Mental Health (R34 MH104470) awarded to Dr. Hanson. Manuscript preparation was supported in part by a National Institute on Alcohol Abuse and Alcoholism T32 Fellowship (T32 AA007459; PI: Monti) awarded to Dr. Helseth.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that we have no conflict of interest.
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee (MUSC Institutional Review Board for Human Research) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Research Involving Human and Animal Rights
This article does not contain any studies with animals performed by any of the authors.
Informed Consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Helseth, S.A., Peer, S.O., Are, F. et al. Sustainment of Trauma-Focused and Evidence-Based Practices Following Learning Collaborative Implementation. Adm Policy Ment Health 47, 569–580 (2020). https://doi.org/10.1007/s10488-020-01024-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-020-01024-3