Departing from the essential features of a high quality systematic review of psychotherapy: A response to Öst (2014) and recommendations for improvement
Introduction
The evidence base for the efficacy of Acceptance and Commitment Therapy (ACT) is substantial. ACT is currently listed on the APA Division 12 website as having strong research support for chronic pain and modest research support for depression, mixed anxiety, obsessive compulsive disorder, and psychosis. The website of the Association for Contextual Behavioral Science (https://contextualscience.org/ACT_Randomized_Controlled_Trials) currently lists 171 randomized trials and several dozens more are as of yet uncatalogued because they exist only in non-English versions. Entering even a short list of ACT relevant subject terms in the Web of Science leads to the identification of well over 1000 articles.
Such a large body of extant research, about 80% which has been produced in the last five years, has led to a series of efforts to summarize and evaluate the ACT and acceptance-based behavior therapy literature and to consider its implications. At least 14 meta-analyses of ACT have appeared since 2009 (see https://contextualscience.org/state_of_the_act_evidence). A recent meta-analysis in the area of anxiety and depression using sequential meta-analytic techniques (Hacker, Stone, & MacBeth, 2016) found that ACT had reached “sufficiency” (i.e. a point at which further research is unlikely to reveal different conclusions) for a large within-treatment effect and a moderate between-group comparison effect in most areas at posttreatment but not superiority over existing evidence-based methods. A-Tjak et al. (2015) and Powers, Zum Vorde Sive Vording, and Emmelkamp (2009) conducted independent meta-analyses and found similar results across a wider range of mental health problems with ACT outperforming control conditions at posttreatment and follow-up for primary outcomes, but with no significant difference from traditional cognitive behavioral therapy (CBT) more generally. Ruiz (2012), in a meta-analysis focused specifically on comparing ACT to CBT, found that ACT outperformed CBT overall, for depression and for quality of life in the studies analyzed. A recent targeted meta-analysis of studies of substance use disorders (Lee, An, Levin, & Twohig, 2015) found that ACT was statistically superior to active treatment comparisons including CBT, but not when CBT alone was considered. Meta-analyses have also shown that the treatment components of the psychological flexibility model (e.g. acceptance, mindfulness, values) underpinning ACT produce positive and sometimes additive effects (Levin, Hildebrandt, Lillis, & Hayes, 2012).
Against this backdrop, a review by Öst (2014) stands out for its conclusions regarding ACT research. Öst (2014) concluded that the average quality of research in ACT was not improving based on a methodological scale that he developed. In contrast to these conclusions, A-Tjak et al.’s (2015) meta-analysis found that ACT research was improving methodologically according to the same scale. In a recent commentary comparing Öst (2014) with A-Tjak et al.’s (2015) analysis, Hertenstein and Nissen (2015, p. 250) suggest: “It is apparent that the two meta-analyses reach strikingly contrasting conclusions, calling for a critical investigation of the potential reasons for this difference.” That is the purpose of the present article.
Gaudiano’s (2009a) re-visiting of Öst’s (2008) original meta-analysis demonstrates that average methodological scores alone do not say much about a research program. The primary question is whether enough high quality studies are available to establish robust scientific conclusions. Methodological ratings thus become most relevant in weighing the additive effects of several studies and their strengths and weaknesses. Such a use of methodological analysis requires very careful attention to the small details. Study-by-study, the ratings need to be relevant, reliable, and examined in detail, rather than in a global or “all-or-none” fashion.
An interest in such details is important in part because Öst (2014) argued broadly that the degree of research evidence for ACT has been systematically over-estimated by the Society of Clinical Psychology (Division 12 of the American Psychological Association) across all disorders it has reviewed. Öst (2014, p. 105) concluded: “ACT is not yet well-established for any disorder.” Web of Science shows that the 2014 meta-analysis has already been cited 56 times (the 2008 review has been cited 203 times). Its conclusion stands in juxtaposition to meta-analyses concluding sufficiency has been reached in some key areas (Hacker et al., 2016), the inclusion of ACT on the Substance Abuse and Mental Health Services Administration's National Registry of Evidence-based Practices and Procedures, and the decision by the U.S. Veterans Administration to deploy ACT as an evidence-based method, and to inclusion on the Division 12 evidence-based therapy list itself for multiple specific conditions.
Scholarly criticism is important in science. Indeed, the society of professionals who are primarily responsible for developing ACT, the Association for Contextual Behavioral Science (ACBS; www.contextualscience.org), has several times had Öst speak about his concerns at ACBS conferences, resulting in useful debate and discussion of the issues. Unfortunately, an examination of the Öst (2014) review suggests that there may have been departures from standard practice for systematic reviews as we detail below. These departures from standard practice appear to have contributed to errors across all sections of Öst's review, and to a variety of conclusions that seem to be objectively unjustified in light of the evidence.
In preparing this response, we first asked all lead study authors to comment on their own studies. We then checked the original papers to verify and confirm possible errors in Öst's (2014) analysis. In most cases the author claims were included in this response. The authors reported errors for 48/60 (80%) of the studies. There were 50 errors in Ost’s (2014) Table 1 alone (6.4% of the total figures reported; see Appendix A) which summarized the methodological specifics of the studies. These were all errors of fact, not interpretation. We have only included errors where the correct facts were reported in the original paper: statements that were shown to be incorrect by additional information that was not in the original manuscript, were counted as being accurately reported. While many of these errors might seem minor if they were just reported in Ost’s (2014) Table 1, the majority of them were against ACT and it seems likely that these errors were also reflected in his meta-analysis and estimates of effect sizes. For example, Öst claimed there was no follow-up data for five studies that in fact did report follow-up data. Presumably, this also meant that incorrect figures were used in the effect size calculations for the meta-analysis (we will explain below why we are using the word “presumably”). The situation appears to be worse for the more interpretive sections of the review such as Ost’s (2014) Tables 11 and 12 (see Appendix B) where we estimate approximately 12% of the reported figures are incorrect. In this area, we found that all of the errors of interpretation were against ACT.
The present article argues that the pattern and magnitude of errors are serious enough that both the content of Öst's (2014) review and the process used to create it should now be set aside in making decisions regarding the treatment efficacy of ACT and in planning further examinations of this literature. The present paper will also briefly discuss the issues surrounding the development of useful criteria for assessing quality of research across different psychotherapeutic traditions, and will note additional criteria that we believe have been minimized or left out. Finally, we will summarize briefly the current state of the evidence for three disorders that have been most intensively studied.
Providing evidence of error is inherently very detailed work. While we will try to be succinct, in order to evaluate the correctness of our conclusions the reader will need to tolerate exposure to details that are important primarily when viewed as an overall pattern. Our intention here is to provide sufficient evidence of the problems so that readers can make their own scientific judgment of the 2014 review and so that future recommendations can be made.
Öst's (2014) review consisted of four parts: (a) selection of studies, (b) evaluation of methodological quality of studies, (c) a meta-analysis and (d) a subjective evaluation of the degree of research evidence for ACT overall and for particular conditions. It is important to be clear on the difference between parts b and d. In part (b), Öst used 22 criteria he developed initially in his 2008 analysis to rate the quality of the studies themselves, whereas in part (d) Öst provided his personal opinions about the APA Division 12 Taskforce criteria for evaluating the quality of evidence for a treatment overall within particular problem areas.
Despite written and face-to-face requests, Öst has not provided us with the actual study by study effect size data used in his meta-analyses. Thus, we have not evaluated his meta-analysis (part c) in this paper. Öst has provided us with his ratings of methodological quality, however, and Ost’s Tables 11 and 12 of his paper (2014) provide nearly complete data for his conclusions regarding the strength of research evidence for specific disorders. Thus, our focus will be on the areas where we have the data needed for a careful examination of the paper: parts a, b and d.
Section snippets
Part a): selection of studies
Öst (2014) clearly describes his criteria for inclusion of studies. We have concerns in a few areas. Unlike A-Tjak et al., 2015, Öst, 2014 included studies with fewer than 10 participants per cell in the design. Larger studies tend to have smaller effect sizes and higher quality ratings (Barth et al., 2013). An examination of smaller studies can make sense if there is a detailed theoretical attempt to explore innovations, to include research from developing nations or from students, or detect
Summary and additional concerns
We have focused this article so far on the factual errors made by Öst. We have not attempted to list the many selective interpretations of data that simply leave out relevant information. In some studies, Öst chose to focus upon the outcome variable that did not change, ignoring clinically crucial outcomes that did improve significantly (Gaudiano and Herbert, 2006, McCracken et al., 2013, Wicksell et al., 2009). In others, he chose to ignore evidence regarding significant reductions in, for
The disciplinary nature of methodological quality standards
We support the development of standards of desirable methodological quality and efforts to summarize the literature in order to make policy recommendations. In our view, however, this needs to be done as a collaborative activity by the discipline itself. Efforts such as the APA Division 12 EBT list or SAMHSA's NREPP program have well specified and collaboratively agreed upon criteria for evaluating research quality and the extent of empirical support.
Meta-analyses should also rely upon diverse
Conclusion
The Öst (2014) review departed from essential features of a high quality systematic review of psychotherapy. Its most fundamental empirical errors are the use of an idiosyncratic and unvalidated rating scheme that appears not to have been reliably applied, as well as numerous factual and interpretive errors in the reporting of trials included in the review. In all areas we could review, quality ratings, facts, and interpretations, errors were dominantly biased against ACT trials. Given these
References (97)
- et al.
Acceptance and commitment therapy for anxiety and OCD spectrum disorders: An empirical review
Journal of Anxiety Disorders
(2014) - et al.
Randomized, controlled pilot trial of a smartphone app for smoking cessation using acceptance and commitment therapy
Drug and Alcohol Dependence
(2014) - et al.
Neural responses to social threat and predictors of cognitive behavioral therapy and acceptance and commitment therapy in social anxiety disorder
Psychiatry Research: Neuroimaging
(2017) - et al.
Acceptance and commitment therapy and the treatment of persons at risk for long-term disability resulting from stress and pain symptoms: A preliminary randomized trial
Behavior Therapy
(2004) - et al.
Physiological and behavioral indices of emotion dysregulation as predictors of outcome from cognitive behavioral therapy and acceptance and commitment therapy for anxiety
Journal of Behavior Therapy and Experimental Psychiatry
(2015) - et al.
Acceptance-based exposure therapy for public speaking anxiety
Journal of Contextual Behavioral Science
(2012) - et al.
Validity of four pain intensity rating scales
PAIN
(2011) - et al.
A randomised worksite comparison of acceptance and commitment therapy and stress inoculation training
Behaviour Research and Therapy
(2010) - et al.
Comparison of acceptance-based and standard cognitive-based coping strategies for craving sweets in overweight and obese women
Eating Behaviors
(2013) Öst's (2008) methodological comparison of clinical trials of acceptance and commitment therapy versus cognitive behavior therapy: Matching Apples with Oranges?
Behav Res Ther
(2009)
Acute treatment of inpatients with psychotic symptoms using Acceptance and Commitment Therapy: Pilot results
Behav Res Ther
Acceptance-based Treatment for Smoking Cessation: Behavior Therapy
Does acceptance and relationship focused behavior therapy contribute to bupropion Outcomes? A randomized controlled trial of functional analytic psychotherapy and acceptance and commitment therapy for smoking cessation
Behavior Therapy
Preliminary data on an acceptance-based emotion regulation group intervention for deliberate self-harm among women with borderline personality disorder
Behavior Therapy
Acceptance and commitment therapy - do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials
Journal of Affective Disorders
Relationship between methamphetamine use history and segmental hair analysis findings of MA users
Forensic Science International
Acceptance and commitment therapy: Model, processes and outcomes
Behav Res Ther
Mindfulness and acceptance-based group therapy versus traditional cognitive behavioral group therapy for social anxiety disorder: A randomized controlled trial
Behav Res Ther
An initial meta-analysis of Acceptance and Commitment Therapy for treating substance use disorders
Drug and Alcohol Dependence
Do we need to challenge thoughts in cognitive behavior therapy?
Clinical Psychology Review
Effectiveness of group acceptance and commitment therapy for fibromyalgia: A 6-month randomized controlled trial (EFFIGACT study)
PAIN
A trial of a brief group-based form of acceptance and commitment therapy (ACT) for chronic pain in general practice: Pilot outcome and process results
Journal of Pain
Public speaking avoidance as a treatment moderator for social anxiety disorder
Journal of Behavior Therapy and Experimental Psychiatry
Acceptance and commitment therapy group treatment for symptoms of borderline personality disorder: A public sector pilot study
Cognitive and Behavioral Practice
Cognitive mediators of treatment for social anxiety Disorder: Comparing acceptance and commitment therapy and cognitive-behavioral therapy
Behavior Therapy
Applying a novel statistical method to advance the personalized treatment of anxiety disorders: A composite moderator of comparative drop-out from CBT and ACT
Behav Res Ther
Efficacy of the third wave of behavioral therapies: A systematic review and meta-analysis
Behav Res Ther
The efficacy of Acceptance and Commitment Therapy: An updated systematic review and meta-analysis
Behav Res Ther
Validity of the life satisfaction questions, the life satisfaction Questionnaire, and the satisfaction with life scale in persons with spinal cord injury
Archives of Physical Medicine and Rehabilitation
A randomised controlled trial of acceptance-based cognitive behavioural therapy for command hallucinations in psychotic disorders
Behav Res Ther
A stage I pilot study of acceptance and commitment therapy for methadone detoxification
Drug Alcohol Depend
Acceptance and commitment therapy in the treatment of alcohol use disorder and comorbid affective disorder: A pilot matched control trial
Behavior Therapy
Psychological flexibility and catastrophizing as associated change mechanisms during online Acceptance & Commitment Therapy for chronic pain
Behaviour Research and Therapy
Core outcome domains for chronic pain clinical trials: IMMPACT recommendations
PAIN
Acceptance and commitment therapy versus tinnitus retraining therapy in the treatment of tinnitus: a randomised controlled trial
Behavior Research and Therapy
Acceptance and commitment therapy for generalized anxiety disorder in older adults: A preliminary report
Behavior Therapy
A randomized, controlled trial of acceptance and commitment therapy and cognitive-behavioral therapy for chronic pain
PAIN
A feasibility study of Acceptance and Commitment Therapy for emotional dysfunction following psychosis
Behaviour Research and Therapy
Evaluating the effectiveness of exposure and acceptance strategies to improve functioning and quality of life in longstanding pediatric pain–a randomized controlled trial
PAIN
A meta-analysis of the efficacy of acceptance and commitment therapy for clinically relevant mental and physical health problems
Psychotherapy and Psychosomatics
The use of the development and well-being assessment (DAWBA) in clinical practice: A randomized trial
European Child & Adolescent Psychiatry
Randomized clinical trial of cognitive behavioral therapy (CBT) versus acceptance and commitment therapy (ACT) for mixed anxiety disorders
Journal of Consulting and Clinical Psychology
Comparative efficacy of seven psychotherapeutic interventions for patients with depression: A network meta-analysis
PLoS Medicine
Nicotine metabolite ratio as a predictor of cigarette consumption
Nicotine & Tobacco Research
A controlled pilot evaluation of acceptance and commitment training for intellectual disability staff
Mindfulness
Mediators of change in emotion-focused and problem-focused worksite stress management interventions
Journal of Occupational Health Psychology
Problems with the use of placebo conditions in psychotherapy research, suggested alternatives, and some strategies for the pursuit of the placebo phenomenon
Journal of Clinical Psychology
Randomized trial of telephone-delivered acceptance and commitment therapy versus cognitive behavioral therapy for smoking cessation: A pilot study
Nicotine & Tobacco Research
Cited by (32)
Evolving an idionomic approach to processes of change: Towards a unified personalized science of human improvement
2022, Behaviour Research and TherapyGroup-based acceptance and commitment therapy interventions for improving general distress and work-related distress in healthcare professionals: A systematic review and meta-analysis
2021, Journal of Affective DisordersCitation Excerpt :The POMFR scale was included because it provides a psychological intervention specific measure of study quality. However, recently, clinical researchers have questioned its validity for assessing the quality of trials of ACT (Atkins et al., 2017). Therefore, in order to ensure maximum rigour, we decided also to use Cochrane Collaboration's Tool, which assesses bias across all types of clinical interventions.
Single-session acceptance and commitment therapy (ACT) interventions for patients with chronic health conditions: A systematic review and meta-analysis
2021, Journal of Contextual Behavioral ScienceScaling up and scaling out: Consilience and the evolution of more nurturing societies
2020, Clinical Psychology ReviewCitation Excerpt :More recently, evidence has accumulated that psychological “disorders” can be largely understood in terms of efforts to avoid threatening or aversive experiences (Hayes, Strosahl, & Wilson, 2016). Clinical research on the treatment of diverse disorders shows that when people are helped to become willing to experience, rather than avoid, psychological or physical aversive sensations and to pursue a set of chosen values, their pain diminishes and the meaning and joy in their lives is enhanced (A-Tjak et al., 2015; Atkins et al., 2017; Jiménez, 2012). Moreover, it appears that when people are encouraged to freely choose their values rather than adopt the values that they perceive others demand that they live by, they typically choose a set of prosocial values that involve caring relationships with others (Gagné, 2003; Sheldon, Arndt, & Houser-Marko, 2003).