Skip to main content

Advertisement

SpringerLink
Log in

Are Randomized Control Trials the Best Method to Assess the Effectiveness of Community Treatment Orders?

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Many jurisdictions have enacted community treatment order (CTO) legislation that requires a person, who suffers from a severe mental disorder, to follow a treatment plan when living in the community. CTOs have been a source of debate because of controversies on whether evidence of effectiveness should only be considered from randomized controlled trials (RCTs). RCTs are considered the “gold standard” method to evaluate effectiveness of simple therapeutic interventions such as medication, but they are problematic for evaluation of complex interventions because valid attribution of causation in complex interventions is not guaranteed with RCTs. CTOs are complex interventions that require the interaction of many individuals and organizations to achieve their effects and effectiveness research must measure these complexities of delivery and outcomes. This paper examines conceptual, methodological and analytical challenges of CTO research within the context of RCTs and other research designs. It also discusses the current state of knowledge on effectiveness of CTOs. Finally, we suggest a way forward by presenting alternative causal inference approaches and potential models for evaluation complex interventions, such as CTOs. We propose that these approaches should be used alongside other research designs in a nuanced approach that may involve using findings from initial studies to refine the intervention and/or its implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Adapted from Vingilis and Pederson (2001) and Bishop and Vingilis (2006)

Similar content being viewed by others

References

  • Amador, X., & Johanson, A.-L. (2000). I am not sick, I don’t need help!. New York: Vida Press.

    Google Scholar 

  • Astbury, B., & Leew, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 30, 363–381. https://doi.org/10.1177/1098214010371972.

    Article  Google Scholar 

  • Barnes, J., Stein, A., & Rosenberg, W. (1999). Evidence based medicine and evaluation of mental health services: Methodological issues and future directions. Archives of Disease in Childhood, 80, 280–285. https://doi.org/10.1136/adc.80.3.280.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Befani, B. (2012). Models of causality and causal inference. In E. Stern, N. Stame, J. Mayne, K. Forss, R. Davies & B. Befani (Eds.), Broadening the range of designs and methods for impact evaluations, DFID Working Paper 38, London: Department for International Development. https://www.oecd.org/derec/50399683.pdf.

  • Benson, K., & Hartz, A. J. (2000). A comparison of observational studies and randomized, controlled trials. New England Journal of Medicine, 342, 1878–1886.

    Article  PubMed  CAS  Google Scholar 

  • Bickman, L., & Reich, S. M. (2009). Random controlled trials a gold standard with feet of clay. In S.I. Donaldson & C. A. Christie (Eds.), What counts as credible evidence in applied research and evaluation practice (pp. 51–72). Thousand Oaks, CA: Sage Publications Inc.

    Chapter  Google Scholar 

  • Bigby, M. (2001). Challenges to the hierarchy of evidence: Does the emperor have no clothes? Archives of Dermatology, 137, 345–356.

    PubMed  CAS  Google Scholar 

  • Biglan, A., Ary, D., & Wagenaar, A. C. (2000). The value of interrupted time-series experiments for community intervention research. Prevention Science, 1, 31–49.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Bishop, J., & Vingilis, E. (2006). Development of a framework for comprehensive evaluation of client outcomes in community mental health services. The Canadian Journal of Program Evaluation, 21, 133–180.

    Google Scholar 

  • Black, N. (1996). Why we need observational studies to evaluate the effectiveness of health care. British Medical Journal, 312, 1215–1218.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Brass, C. T., Nunez-Neto, B., & Williams, E. D. (2006). Congress and program evaluation: an overview of randomized controlled trials (RCTs) and related issues. Washington, DC: The Library of Congress. http://digital.library.unt.edu/ark:/67531/metacrs9145/m1/1/high_res_d/RL33301_2006Mar07.pdf.

  • Burns, T., Rugkåsa, J., Molodynski, A., Dawson, J., Yeeles, K., & Vazquez-Montes M., et al. (2013). Community treatment orders for patients with psychosis (OCTET): a randomised controlled trial. Lancet, 381, 1627–1633. https://doi.org/10.1016/S0140-6736(13)60107-5.

    Article  PubMed  Google Scholar 

  • Chen, H.-T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage Publications Inc.

    Google Scholar 

  • Chen, H.-T. (2005). Practical program evaluation: assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage Publications Inc.

    Book  Google Scholar 

  • Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies and the hierarchy of research designs. New England Journal of Medicine, 342, 1887–1892.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation design & analysis issues for field settings. Boston: Houghton Mifflin Co.

    Google Scholar 

  • Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750.

    Article  Google Scholar 

  • Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: The new Medical Research Council guidance. British Medical Journal, 337, a1655. https://doi.org/10.1136/bmj.a1655.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dale, E. (2010). Is supervised community treatment ethically justified? Journal of Medical Ethics, 36, 271–274.

    Article  PubMed  Google Scholar 

  • Dawson, J. (2016). Doubts about the clinical effectiveness of community treatment orders. Canadian Journal of Psychiatry, 61, 4–6.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dawson, J., Burns, T., & Rugkåsa, J. (2011). Lawfulness of a randomised trial of the new community treatment order regime for England and Wales. Medical Law Review, 19, 1–26.

    Article  PubMed  CAS  Google Scholar 

  • DeRidder, R., Molodynski, A., Manning, C., McCusker, P., & Rugkåsa, J. (2016). Community treatment orders in the UK 5 years on: a repeat national survey of psychiatrists. Psychiatric Bulletin, 40, 119–123

    Google Scholar 

  • Donaldson, S.I. (2007). Program theory-driven evaluation science. New York, NY: Laurence Ehlbaum Assoc.

    Google Scholar 

  • Faries, D. E., Nyhuis, A. W., & Ascher-Svanum, H. (2009). Methodological issues in assessing changes in costs pre- and post-medication switch: A schizophrenia study example. Cost Effectiveness and Resource Allocation. https://doi.org/10.1186/1478-7547-7-11.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fernandez, G. A., & Nygard, S. (1990). Impact of involuntary outpatient commitment on the revolving-door syndrome in North Carolina. Hospital and Community Psychiatry, 41, 1001–1004.

    PubMed  CAS  Google Scholar 

  • Fletcher, A., Jamal, F., Moore, G., Evans, R. E., Murphy, S., & Bonell, C. (2016). Realist complex intervention science: Applying principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation, 22, 286–303.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fonagy, P. (1999). Process and outcome in mental health care delivery: A model approach to treatment evaluation. Bulletin of the Menninger Clinic, 63, 288–304.

    PubMed  CAS  Google Scholar 

  • Government of Canada. (2012). Theory-based appraches to evaluation: concepts and practices. https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/centre-excellence-evaluation/theory-based-approaches-evaluation-concepts-practices.html.

  • Grembowski, D. (2001). The practice of health program evaluation. Thousand Oaks, CA: Sage Publications Inc.

    Book  Google Scholar 

  • Hasson, H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Science, 5, 1–9.

    Article  Google Scholar 

  • Hawe, P., Shiell, A., & Riley, T. (2004). Complex interventions: how “out of control” can a randomised controlled trial be? British Medical Journal, 328, 1363–1561.

    Article  Google Scholar 

  • Hawkins, A. J. (2016). Realist evaluation and randomised controlled trials for testing program theory in complex social systems. Evaluation, 22, 270–285.

    Article  Google Scholar 

  • Hohmann, A. A., & Shear, M. K. (2002). Community-based intervention research: Coping with the “noise” of real life in study design. American Journal of Psychiatry, 159, 201–207.

    Article  PubMed  Google Scholar 

  • Imai, K., Keele, K., Tingley, D., & Yamamoto, T. (2011). Unpacking the black box of causality: Leaning about causal mechanisms from experimental and observational studies. American Political Science Review, 105, 765–789.

    Article  Google Scholar 

  • Kisely, S., Preston, N., Xiao, J., Lawrence, D., Louise, S., & Crowe, E. (2013). Reducing all-cause mortality among patients with psychiatric disorders: a population-based study. Canadian Medical Association Journal, 185, E50–E56. https://doi.org/10.1503/cmaj.121077.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lepping, P., & Malik, M. (2013). Community treatment orders: current practice and a framework to aid clinicians. The Psychiatrist, 37, 54–57.

    Article  Google Scholar 

  • Lipsey, M. W. (1988). Practice and malpractice in evaluation research. Evaluation Practice, 9, 5–23.

    Article  Google Scholar 

  • Lipsey, M. W., & Cordray, D. S. (2000). Evaluation methods for social intervention. Annual Review of Psychology, 51, 345–375.

    Article  PubMed  CAS  Google Scholar 

  • McPherson, K. (1994). The best and the enemy of the good - randomized controlled trials, uncertainty, and assessing the role of patient choice in medical decision-making. Journal of Epidemiology and Community Health, 48, 6–15.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • National Association of State Mental Health Program Directors (NASMHPD) (2001). Technical report on involuntary outpatient commitment. Alexandria, VA: Mental Health Services, Substance Abuse and Mental Health Services Administration, U.S. Department of Health and Human Services.

    Google Scholar 

  • Nelson, G., Aubry, T., & Lafrance, A. (2007). A review of the literature of the effectiveness of housing and support, assertive community treatment, and intensive case management interventions for persons with mental illness who have been homeless. American Journal of Othopsychistry, 77, 350–361.

    Article  Google Scholar 

  • O’Reilly, R. (2011). Research on community treatment orders. In E. R. Vingilis, & S. A. State (Eds.), Applied research and evaluation in community mental health services (pp. 68–79). Montreal: McGill-Queen’s University Press.

    Google Scholar 

  • O’Reilly, R., Corring, D., Richard, J., Plyley, C., & Pallaveshi, L. (2016). Do intensive services obviate the need for CTOs? International Journal of Law and Psychiatry, 47, 74–78.

    Article  PubMed  Google Scholar 

  • O’Reilly, R. L. (2004). Why are community treatment orders controversial? Canadian Journal of Psychiatry, 49, 579–584.

    Article  PubMed  Google Scholar 

  • Oakley, A., Strange, V., Bonell, C., Allen, E., & Stephenson, J. & RIPPLE Study Team. (2006). Process evaluation in randomised controlled trials of complex interventions. British Medical Journal, 332, 413–416.

    Article  PubMed  PubMed Central  Google Scholar 

  • Pawson, R. (2013). The science of evaluation: A realist manifesto. London: Sage Publications.

    Book  Google Scholar 

  • Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review: A new method of systematic review designed for complex policy interventions. Journal of Health Services Research and Policy, 10, 21–34.

    Article  PubMed  Google Scholar 

  • Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage Publications.

    Google Scholar 

  • Pawson, R., Wong, G., & Owen, L. (2011). Known knowns, known unknowns, unknowns unknowns: A predicament of evidence-based policy. American Journal of Evaluation, 32, 518–546. https://doi.org/10.1177/109821403831.

    Article  Google Scholar 

  • Pearson, C., Montgomery, A. E., & Locke, G. (2009). Housing stability among homeless individuals with serious mental illness participating in housing first programs. Journal of Community Psychology, 37, 404–417.

    Article  Google Scholar 

  • Persons, J. B., & Silberschatz, G. (1998). Are results of randomized controlled trials useful to psychotherapists? Journal of Consulting and Clinical Psychology, 66, 126–135.

    Article  PubMed  CAS  Google Scholar 

  • Posavac, M. J., & Carey, R. G. (2003). Program evaluation methods and case studies. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Preston, N. J., Kisely, S., & Xiao, J. (2002). Assessing the outcome of compulsory psychiatric treatment in the community: epidemiological study in Western Australia. British Medical Journal, 324, 1244. https://doi.org/10.1136/bmj.324.7348.1244.

    Article  PubMed  PubMed Central  Google Scholar 

  • Reichardt, C. S. (2011). Evaluating methods for estimating program effects. American Journal of Evaluation, 32, 246–272. https://doi.org/10.1177/1098214011398954.

    Article  Google Scholar 

  • Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th edn.). Thousand Oaks, CA: Sage Publications Inc.

    Google Scholar 

  • Rugkåsa, J. (2016). Effectiveness of community treatment orders: the international evidence. Canadian Journal of Psychiatry, 61, 15–24.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rugkåsa, J., Molodynski, A., Yeeles, K., Vazquez Montes, M., Visser, C., Burns, T. & OCTET Group. (2015). Community treatment orders: clinical and social outcomes, and a subgroup analysis from the OCTET RCT. Acta Psychiatrica Scandinavica, 131, 321–329.

    Article  PubMed  Google Scholar 

  • Scriven, M. (2015). Demythologizing causation and evidence. In S.I. Donaldson, C. A. Christie & M. M. Mark (Eds.), Credible and actionable evidence the foundation for rigorous and influential evaluations (2nd edn.). Thousand Oaks, CA: Sage Publications Inc.

    Google Scholar 

  • Segal, S. P., & Burgess, P. M. (2006). Effect of conditional release from hospitalization on mortality risk. Psychiatric Services, 57, 1607–1613.

    Article  PubMed  Google Scholar 

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Co.

    Google Scholar 

  • Smith, M., Branton, T., & Cardno, A. (2014). Is the bark worse than the bite? Additional conditions used within community treatment orders. Psychiatric Bulletin, 38, 9–12.

    Article  Google Scholar 

  • Steadman, H. J., Gounis, K., Dennis, D., Hopper, K., Roche, B., Swartz, M., & Robbins, P. C. (2001). Assessing the New York City involuntary outpatient commitment pilot program. Psychiatric Services, 52, 330–336.

    Article  PubMed  CAS  Google Scholar 

  • Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (Eds.) (2012). Broadening the range of designs and methods for impact evaluations, DFID Working Paper 38. London: Department for International Development. https://www.oecd.org/derec/50399683.pdf.

  • Swanson, J. W., & Swartz, M. S. (2014). Why the evidence for outpatient commitment is good enough. Psychiatric Services, 65, 808–811. https://doi.org/10.1176/appi.ps.201300424.

    Article  PubMed  Google Scholar 

  • Swartz, M. S., Swanson, J. W., Wagner, H. R., Burns, B. J., Hiday, V. A., & Borum, R. (1999). Can involuntary outpatient commitment reduce hospital recidivism?: Findings from a randomized trial with severely mentally ill individuals. American Journal of Psychiatry, 156, 1968–1975.

    PubMed  CAS  Google Scholar 

  • Swartz, M. S., Wilder, C. M., Swanson, J. W., Van Dorn, R. A., Robbins, P. C., Steadman, H. J., … Monahan, J. (2010). Assessing outcomes for consumers in New York’s assisted outpatient treatment program. Psychiatric Services, 61, 976–981.

    Article  PubMed  Google Scholar 

  • Sweetman, E., & Doig, G. S. (2011). Failure to report protocol violations in clinical trials: a threat to internal validity?. Trials, 12, 214.

    Article  PubMed  PubMed Central  Google Scholar 

  • Taylor, M., Macpherson, M., Macleod, C., & Lyons, D. (2016). Community treatment orders and reduced time in hospital: a nationwide study, 2007–2012. British Journal of Psychiatry Bulletin, 40, 124–126.

    Google Scholar 

  • Tones, K. (1997). Beyond the randomized controlled trial: A case for ‘judicial review’. Health Education Research, 12, i–iv.

    Article  PubMed  CAS  Google Scholar 

  • Torrey, E. F. (1997). Out of the shadows: confronting America’s mental illness crisis. New York, NY: Wiley.

    Google Scholar 

  • Vingilis, E. (2011). Issues and challenges in community mental health services program and policy evaluation. In E. R. Vingilis & S. A. State (Eds.), Applied research and evaluation in community mental health services. Montreal: McGill-Queen’s University Press.

    Google Scholar 

  • Vingilis, E., Bishop, J. E. H., He, W., Yue, L. H., Braun, J., Seeley, J., … Mitchell, B. (2015). Effects of client perceptions of therapeutic alliance, empowerment, attitudes toward medication and insight on functioning. International Journal of Clinical Psychiatry and Mental Health, 3, 54–65.

    Article  Google Scholar 

  • Vingilis, E., & Pederson, L. (2001). Using the right tools to answer the right questions: the importance of evaluative research techniques for health services evaluation research in the 21st century. Canadian Journal of Program Evaluation, 16, 1–26.

    Google Scholar 

  • Walsh, D. (2010). Do we need community treatment orders in Ireland? Irish Journal of Psychological Medicine, 27, 90–96.

    Article  PubMed  Google Scholar 

  • Wandersman, A. (2003). Community science: bridging the gap between science and practice with community-centered models. American Journal of Community Psychology, 31, 227–242.

    Article  PubMed  Google Scholar 

  • Weiss, C. J. (1998). Evaluation. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Zanni, G. R., & Stavis, P. F. (2007). The effectiveness and ethical justification of psychiatric outpatient commitment. American Journal of Bioethics, 7, 31–41.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychiatry, Schulich School of Medicine & Dentistry, Western University, London, ON, Canada

    Richard O’Reilly

  2. St. Joseph’s Health Care London, Stn B, P.O. Box 5777, London, ON, N6A 4V2, Canada

    Richard O’Reilly

  3. Departments of Family Medicine and Epidemiology & Biostatistics, Schulich School of Medicine & Dentistry, Western University, London, ON, Canada

    Evelyn Vingilis

Authors
  1. Richard O’Reilly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Evelyn Vingilis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard O’Reilly.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest related to this article.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

O’Reilly, R., Vingilis, E. Are Randomized Control Trials the Best Method to Assess the Effectiveness of Community Treatment Orders?. Adm Policy Ment Health 45, 565–574 (2018). https://doi.org/10.1007/s10488-017-0845-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-017-0845-7

Keywords

Search

Navigation