Elsevier

Behaviour Research and Therapy

Volume 97, October 2017, Pages 259-272
Behaviour Research and Therapy

Departing from the essential features of a high quality systematic review of psychotherapy: A response to Öst (2014) and recommendations for improvement

https://doi.org/10.1016/j.brat.2017.05.016Get rights and content

Highlights

  • We found errors in Öst’s (2014) review of Acceptance and Commitment Therapy (ACT).

  • Öst (2014) should be set aside in future considerations of the evidence for ACT.

  • We suggest how balanced reviews within transdiagnostic frameworks can be conducted.

Abstract

Öst's (2014) systematic review and meta-analysis of Acceptance and Commitment Therapy (ACT) has received wide attention. On the basis of his review, Öst argued that ACT research was not increasing in its quality and that, in contradiction to the views of Division 12 of the American Psychological Association (APA), ACT is “not yet well-established for any disorder” (2014, p. 105). We conducted a careful examination of the methods, approach, and data used in the meta-analysis. Based in part on examinations by the authors of the studies involved, which were then independently checked, 91 factual or interpretive errors were documented, touching upon 80% of the studies reviewed. Comparisons of Öst's quality ratings with independent teams rating the same studies with the same scale suggest that Ost's ratings were unreliable. In all of these areas (factual errors; interpretive errors; quality ratings) mistakes and differences were not random: Ost's data were dominantly more negative toward ACT. The seriousness, range, and distribution of errors, and a wider pattern of misinterpreting the purpose of studies and ignoring positive results, suggest that Öst's review should be set aside in future considerations of the evidence base for ACT. We argue that future published reviews and meta-analyses should rely upon diverse groups of scholars rather than a single individual; that resulting raw data should be made available for inspection and independent analysis; that well-crafted committees rather than individuals should design, apply and interpret quality criteria; that the intent of transdiagnostic studies need to be more seriously considered as the field shifts away from a purely syndromal approach; and that data that demonstrate theoretically consistent mediating processes should be given greater weight in evaluating specific interventions. Finally, in order to examine substantive progress since Öst's review, recent outcome and process evidence was briefly examined.

Introduction

The evidence base for the efficacy of Acceptance and Commitment Therapy (ACT) is substantial. ACT is currently listed on the APA Division 12 website as having strong research support for chronic pain and modest research support for depression, mixed anxiety, obsessive compulsive disorder, and psychosis. The website of the Association for Contextual Behavioral Science (https://contextualscience.org/ACT_Randomized_Controlled_Trials) currently lists 171 randomized trials and several dozens more are as of yet uncatalogued because they exist only in non-English versions. Entering even a short list of ACT relevant subject terms in the Web of Science leads to the identification of well over 1000 articles.

Such a large body of extant research, about 80% which has been produced in the last five years, has led to a series of efforts to summarize and evaluate the ACT and acceptance-based behavior therapy literature and to consider its implications. At least 14 meta-analyses of ACT have appeared since 2009 (see https://contextualscience.org/state_of_the_act_evidence). A recent meta-analysis in the area of anxiety and depression using sequential meta-analytic techniques (Hacker, Stone, & MacBeth, 2016) found that ACT had reached “sufficiency” (i.e. a point at which further research is unlikely to reveal different conclusions) for a large within-treatment effect and a moderate between-group comparison effect in most areas at posttreatment but not superiority over existing evidence-based methods. A-Tjak et al. (2015) and Powers, Zum Vorde Sive Vording, and Emmelkamp (2009) conducted independent meta-analyses and found similar results across a wider range of mental health problems with ACT outperforming control conditions at posttreatment and follow-up for primary outcomes, but with no significant difference from traditional cognitive behavioral therapy (CBT) more generally. Ruiz (2012), in a meta-analysis focused specifically on comparing ACT to CBT, found that ACT outperformed CBT overall, for depression and for quality of life in the studies analyzed. A recent targeted meta-analysis of studies of substance use disorders (Lee, An, Levin, & Twohig, 2015) found that ACT was statistically superior to active treatment comparisons including CBT, but not when CBT alone was considered. Meta-analyses have also shown that the treatment components of the psychological flexibility model (e.g. acceptance, mindfulness, values) underpinning ACT produce positive and sometimes additive effects (Levin, Hildebrandt, Lillis, & Hayes, 2012).

Against this backdrop, a review by Öst (2014) stands out for its conclusions regarding ACT research. Öst (2014) concluded that the average quality of research in ACT was not improving based on a methodological scale that he developed. In contrast to these conclusions, A-Tjak et al.’s (2015) meta-analysis found that ACT research was improving methodologically according to the same scale. In a recent commentary comparing Öst (2014) with A-Tjak et al.’s (2015) analysis, Hertenstein and Nissen (2015, p. 250) suggest: “It is apparent that the two meta-analyses reach strikingly contrasting conclusions, calling for a critical investigation of the potential reasons for this difference.” That is the purpose of the present article.

Gaudiano’s (2009a) re-visiting of Öst’s (2008) original meta-analysis demonstrates that average methodological scores alone do not say much about a research program. The primary question is whether enough high quality studies are available to establish robust scientific conclusions. Methodological ratings thus become most relevant in weighing the additive effects of several studies and their strengths and weaknesses. Such a use of methodological analysis requires very careful attention to the small details. Study-by-study, the ratings need to be relevant, reliable, and examined in detail, rather than in a global or “all-or-none” fashion.

An interest in such details is important in part because Öst (2014) argued broadly that the degree of research evidence for ACT has been systematically over-estimated by the Society of Clinical Psychology (Division 12 of the American Psychological Association) across all disorders it has reviewed. Öst (2014, p. 105) concluded: “ACT is not yet well-established for any disorder.” Web of Science shows that the 2014 meta-analysis has already been cited 56 times (the 2008 review has been cited 203 times). Its conclusion stands in juxtaposition to meta-analyses concluding sufficiency has been reached in some key areas (Hacker et al., 2016), the inclusion of ACT on the Substance Abuse and Mental Health Services Administration's National Registry of Evidence-based Practices and Procedures, and the decision by the U.S. Veterans Administration to deploy ACT as an evidence-based method, and to inclusion on the Division 12 evidence-based therapy list itself for multiple specific conditions.

Scholarly criticism is important in science. Indeed, the society of professionals who are primarily responsible for developing ACT, the Association for Contextual Behavioral Science (ACBS; www.contextualscience.org), has several times had Öst speak about his concerns at ACBS conferences, resulting in useful debate and discussion of the issues. Unfortunately, an examination of the Öst (2014) review suggests that there may have been departures from standard practice for systematic reviews as we detail below. These departures from standard practice appear to have contributed to errors across all sections of Öst's review, and to a variety of conclusions that seem to be objectively unjustified in light of the evidence.

In preparing this response, we first asked all lead study authors to comment on their own studies. We then checked the original papers to verify and confirm possible errors in Öst's (2014) analysis. In most cases the author claims were included in this response. The authors reported errors for 48/60 (80%) of the studies. There were 50 errors in Ost’s (2014) Table 1 alone (6.4% of the total figures reported; see Appendix A) which summarized the methodological specifics of the studies. These were all errors of fact, not interpretation. We have only included errors where the correct facts were reported in the original paper: statements that were shown to be incorrect by additional information that was not in the original manuscript, were counted as being accurately reported. While many of these errors might seem minor if they were just reported in Ost’s (2014) Table 1, the majority of them were against ACT and it seems likely that these errors were also reflected in his meta-analysis and estimates of effect sizes. For example, Öst claimed there was no follow-up data for five studies that in fact did report follow-up data. Presumably, this also meant that incorrect figures were used in the effect size calculations for the meta-analysis (we will explain below why we are using the word “presumably”). The situation appears to be worse for the more interpretive sections of the review such as Ost’s (2014) Tables 11 and 12 (see Appendix B) where we estimate approximately 12% of the reported figures are incorrect. In this area, we found that all of the errors of interpretation were against ACT.

The present article argues that the pattern and magnitude of errors are serious enough that both the content of Öst's (2014) review and the process used to create it should now be set aside in making decisions regarding the treatment efficacy of ACT and in planning further examinations of this literature. The present paper will also briefly discuss the issues surrounding the development of useful criteria for assessing quality of research across different psychotherapeutic traditions, and will note additional criteria that we believe have been minimized or left out. Finally, we will summarize briefly the current state of the evidence for three disorders that have been most intensively studied.

Providing evidence of error is inherently very detailed work. While we will try to be succinct, in order to evaluate the correctness of our conclusions the reader will need to tolerate exposure to details that are important primarily when viewed as an overall pattern. Our intention here is to provide sufficient evidence of the problems so that readers can make their own scientific judgment of the 2014 review and so that future recommendations can be made.

Öst's (2014) review consisted of four parts: (a) selection of studies, (b) evaluation of methodological quality of studies, (c) a meta-analysis and (d) a subjective evaluation of the degree of research evidence for ACT overall and for particular conditions. It is important to be clear on the difference between parts b and d. In part (b), Öst used 22 criteria he developed initially in his 2008 analysis to rate the quality of the studies themselves, whereas in part (d) Öst provided his personal opinions about the APA Division 12 Taskforce criteria for evaluating the quality of evidence for a treatment overall within particular problem areas.

Despite written and face-to-face requests, Öst has not provided us with the actual study by study effect size data used in his meta-analyses. Thus, we have not evaluated his meta-analysis (part c) in this paper. Öst has provided us with his ratings of methodological quality, however, and Ost’s Tables 11 and 12 of his paper (2014) provide nearly complete data for his conclusions regarding the strength of research evidence for specific disorders. Thus, our focus will be on the areas where we have the data needed for a careful examination of the paper: parts a, b and d.

Section snippets

Part a): selection of studies

Öst (2014) clearly describes his criteria for inclusion of studies. We have concerns in a few areas. Unlike A-Tjak et al., 2015, Öst, 2014 included studies with fewer than 10 participants per cell in the design. Larger studies tend to have smaller effect sizes and higher quality ratings (Barth et al., 2013). An examination of smaller studies can make sense if there is a detailed theoretical attempt to explore innovations, to include research from developing nations or from students, or detect

Summary and additional concerns

We have focused this article so far on the factual errors made by Öst. We have not attempted to list the many selective interpretations of data that simply leave out relevant information. In some studies, Öst chose to focus upon the outcome variable that did not change, ignoring clinically crucial outcomes that did improve significantly (Gaudiano and Herbert, 2006, McCracken et al., 2013, Wicksell et al., 2009). In others, he chose to ignore evidence regarding significant reductions in, for

The disciplinary nature of methodological quality standards

We support the development of standards of desirable methodological quality and efforts to summarize the literature in order to make policy recommendations. In our view, however, this needs to be done as a collaborative activity by the discipline itself. Efforts such as the APA Division 12 EBT list or SAMHSA's NREPP program have well specified and collaboratively agreed upon criteria for evaluating research quality and the extent of empirical support.

Meta-analyses should also rely upon diverse

Conclusion

The Öst (2014) review departed from essential features of a high quality systematic review of psychotherapy. Its most fundamental empirical errors are the use of an idiosyncratic and unvalidated rating scheme that appears not to have been reliably applied, as well as numerous factual and interpretive errors in the reporting of trials included in the review. In all areas we could review, quality ratings, facts, and interpretations, errors were dominantly biased against ACT trials. Given these

References (97)

  • B.A. Gaudiano et al.

    Acute treatment of inpatients with psychotic symptoms using Acceptance and Commitment Therapy: Pilot results

    Behav Res Ther

    (2006)
  • E.V. Gifford et al.

    Acceptance-based Treatment for Smoking Cessation: Behavior Therapy

    (2004)
  • E.V. Gifford et al.

    Does acceptance and relationship focused behavior therapy contribute to bupropion Outcomes? A randomized controlled trial of functional analytic psychotherapy and acceptance and commitment therapy for smoking cessation

    Behavior Therapy

    (2011)
  • K.L. Gratz et al.

    Preliminary data on an acceptance-based emotion regulation group intervention for deliberate self-harm among women with borderline personality disorder

    Behavior Therapy

    (2006)
  • T. Hacker et al.

    Acceptance and commitment therapy - do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials

    Journal of Affective Disorders

    (2016)
  • E. Han et al.

    Relationship between methamphetamine use history and segmental hair analysis findings of MA users

    Forensic Science International

    (2015)
  • S.C. Hayes et al.

    Acceptance and commitment therapy: Model, processes and outcomes

    Behav Res Ther

    (2006)
  • N.L. Kocovski et al.

    Mindfulness and acceptance-based group therapy versus traditional cognitive behavioral group therapy for social anxiety disorder: A randomized controlled trial

    Behav Res Ther

    (2013)
  • E.B. Lee et al.

    An initial meta-analysis of Acceptance and Commitment Therapy for treating substance use disorders

    Drug and Alcohol Dependence

    (2015)
  • R.J. Longmore et al.

    Do we need to challenge thoughts in cognitive behavior therapy?

    Clinical Psychology Review

    (2007)
  • J.V. Luciano et al.

    Effectiveness of group acceptance and commitment therapy for fibromyalgia: A 6-month randomized controlled trial (EFFIGACT study)

    PAIN

    (2014)
  • L.M. McCracken et al.

    A trial of a brief group-based form of acceptance and commitment therapy (ACT) for chronic pain in general practice: Pilot outcome and process results

    Journal of Pain

    (2013)
  • B. Mesri et al.

    Public speaking avoidance as a treatment moderator for social anxiety disorder

    Journal of Behavior Therapy and Experimental Psychiatry

    (2017)
  • J. Morton et al.

    Acceptance and commitment therapy group treatment for symptoms of borderline personality disorder: A public sector pilot study

    Cognitive and Behavioral Practice

    (2012)
  • A.N. Niles et al.

    Cognitive mediators of treatment for social anxiety Disorder: Comparing acceptance and commitment therapy and cognitive-behavioral therapy

    Behavior Therapy

    (2014)
  • A.N. Niles et al.

    Applying a novel statistical method to advance the personalized treatment of anxiety disorders: A composite moderator of comparative drop-out from CBT and ACT

    Behav Res Ther

    (2017)
  • L.-G. Öst

    Efficacy of the third wave of behavioral therapies: A systematic review and meta-analysis

    Behav Res Ther

    (2008)
  • L.-G. Öst

    The efficacy of Acceptance and Commitment Therapy: An updated systematic review and meta-analysis

    Behav Res Ther

    (2014)
  • M.W. Post et al.

    Validity of the life satisfaction questions, the life satisfaction Questionnaire, and the satisfaction with life scale in persons with spinal cord injury

    Archives of Physical Medicine and Rehabilitation

    (2012)
  • F. Shawyer et al.

    A randomised controlled trial of acceptance-based cognitive behavioural therapy for command hallucinations in psychotic disorders

    Behav Res Ther

    (2012)
  • A.L. Stotts et al.

    A stage I pilot study of acceptance and commitment therapy for methadone detoxification

    Drug Alcohol Depend

    (2012)
  • T.B. Thekiso et al.

    Acceptance and commitment therapy in the treatment of alcohol use disorder and comorbid affective disorder: A pilot matched control trial

    Behavior Therapy

    (2015)
  • H.R. Trompetter et al.

    Psychological flexibility and catastrophizing as associated change mechanisms during online Acceptance & Commitment Therapy for chronic pain

    Behaviour Research and Therapy

    (2015)
  • D.C. Turk et al.

    Core outcome domains for chronic pain clinical trials: IMMPACT recommendations

    PAIN

    (2003)
  • V.Z. Westin et al.

    Acceptance and commitment therapy versus tinnitus retraining therapy in the treatment of tinnitus: a randomised controlled trial

    Behavior Research and Therapy

    (2011)
  • J.L. Wetherell et al.

    Acceptance and commitment therapy for generalized anxiety disorder in older adults: A preliminary report

    Behavior Therapy

    (2011)
  • J.L. Wetherell et al.

    A randomized, controlled trial of acceptance and commitment therapy and cognitive-behavioral therapy for chronic pain

    PAIN

    (2011)
  • R. White et al.

    A feasibility study of Acceptance and Commitment Therapy for emotional dysfunction following psychosis

    Behaviour Research and Therapy

    (2011)
  • R.K. Wicksell et al.

    Evaluating the effectiveness of exposure and acceptance strategies to improve functioning and quality of life in longstanding pediatric pain–a randomized controlled trial

    PAIN

    (2009)
  • J.G.L. A-Tjak et al.

    A meta-analysis of the efficacy of acceptance and commitment therapy for clinically relevant mental and physical health problems

    Psychotherapy and Psychosomatics

    (2015)
  • M. Aebi et al.

    The use of the development and well-being assessment (DAWBA) in clinical practice: A randomized trial

    European Child & Adolescent Psychiatry

    (2012)
  • J.J. Arch et al.

    Randomized clinical trial of cognitive behavioral therapy (CBT) versus acceptance and commitment therapy (ACT) for mixed anxiety disorders

    Journal of Consulting and Clinical Psychology

    (2012)
  • J. Barth et al.

    Comparative efficacy of seven psychotherapeutic interventions for patients with depression: A network meta-analysis

    PLoS Medicine

    (2013)
  • N.L. Benowitz et al.

    Nicotine metabolite ratio as a predictor of cigarette consumption

    Nicotine & Tobacco Research

    (2003)
  • J.S. Bethay et al.

    A controlled pilot evaluation of acceptance and commitment training for intellectual disability staff

    Mindfulness

    (2013)
  • F.W. Bond et al.

    Mediators of change in emotion-focused and problem-focused worksite stress management interventions

    Journal of Occupational Health Psychology

    (2000)
  • T.D. Borkovec et al.

    Problems with the use of placebo conditions in psychotherapy research, suggested alternatives, and some strategies for the pursuit of the placebo phenomenon

    Journal of Clinical Psychology

    (2005)
  • J.B. Bricker et al.

    Randomized trial of telephone-delivered acceptance and commitment therapy versus cognitive behavioral therapy for smoking cessation: A pilot study

    Nicotine & Tobacco Research

    (2014)
  • Cited by (32)

    • Group-based acceptance and commitment therapy interventions for improving general distress and work-related distress in healthcare professionals: A systematic review and meta-analysis

      2021, Journal of Affective Disorders
      Citation Excerpt :

      The POMFR scale was included because it provides a psychological intervention specific measure of study quality. However, recently, clinical researchers have questioned its validity for assessing the quality of trials of ACT (Atkins et al., 2017). Therefore, in order to ensure maximum rigour, we decided also to use Cochrane Collaboration's Tool, which assesses bias across all types of clinical interventions.

    • Scaling up and scaling out: Consilience and the evolution of more nurturing societies

      2020, Clinical Psychology Review
      Citation Excerpt :

      More recently, evidence has accumulated that psychological “disorders” can be largely understood in terms of efforts to avoid threatening or aversive experiences (Hayes, Strosahl, & Wilson, 2016). Clinical research on the treatment of diverse disorders shows that when people are helped to become willing to experience, rather than avoid, psychological or physical aversive sensations and to pursue a set of chosen values, their pain diminishes and the meaning and joy in their lives is enhanced (A-Tjak et al., 2015; Atkins et al., 2017; Jiménez, 2012). Moreover, it appears that when people are encouraged to freely choose their values rather than adopt the values that they perceive others demand that they live by, they typically choose a set of prosocial values that involve caring relationships with others (Gagné, 2003; Sheldon, Arndt, & Houser-Marko, 2003).

    View all citing articles on Scopus
    View full text