Evaluating Digital Health Interventions: Key Questions and Approaches

https://doi.org/10.1016/j.amepre.2016.06.008Get rights and content

Digital health interventions have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety, and personalization. Achieving these improvements requires a cumulative knowledge base to inform development and deployment of digital health interventions. However, evaluations of digital health interventions present special challenges. This paper aims to examine these challenges and outline an evaluation strategy in terms of the research questions needed to appraise such interventions. As they are at the intersection of biomedical, behavioral, computing, and engineering research, methods drawn from all of these disciplines are required. Relevant research questions include defining the problem and the likely benefit of the digital health intervention, which in turn requires establishing the likely reach and uptake of the intervention, the causal model describing how the intervention will achieve its intended benefit, key components, and how they interact with one another, and estimating overall benefit in terms of effectiveness, cost effectiveness, and harms. Although RCTs are important for evaluation of effectiveness and cost effectiveness, they are best undertaken only when: (1) the intervention and its delivery package are stable; (2) these can be implemented with high fidelity; and (3) there is a reasonable likelihood that the overall benefits will be clinically meaningful (improved outcomes or equivalent outcomes at lower cost). Broadening the portfolio of research questions and evaluation methods will help with developing the necessary knowledge base to inform decisions on policy, practice, and research.

Introduction

There is enormous potential for digital health interventions (DHIs; i.e., interventions delivered via digital technologies such as smartphones, website, or text messaging) to provide effective, cost effective, safe, and scalable interventions to improve health and healthcare. DHIs can be used to promote healthy behaviors (e.g., smoking cessation,1 healthy eating,2 physical activity,3 safer sex,4 or alcohol consumption5); improve outcomes in people with long-term conditions6 such as cardiovascular disease,7 diabetes,8 and mental health conditions9; and provide remote access to effective treatments (e.g., computerized cognitive behavioral therapy for mental health and somatic problems).10, 11, 12, 13 They are typically complex interventions with multiple components, and many have multiple aims, including enabling users to be better informed about their health, share experiences with others in similar positions, change perceptions and cognitions around health, assess and monitor specified health states or health behaviors, titrate medication, clarify health priorities and reach treatment decisions congruent with these, and improve communication between patients and healthcare professionals (HCPs). Active components may include information, psycho-education, personal stories, formal decision aids, behavior change support, interactions with HCPs and other patients, self-assessment or monitoring tools (questionnaires, wearables, monitors), and effective theory-based psychological interventions developed for face-to-face delivery, such as cognitive behavioral therapy or mindfulness training.

To date, the potential of DHIs has scarcely been realized, partly because of difficulties in generating an accumulating knowledge base for guiding decisions about DHIs. These include the rapid change of the wider technology landscape,14 which requires DHIs to constantly evolve and be updated just to remain useful, let alone improve. For example, imagine an iPhone app promoting physical activity, with development and evaluation starting in 2008. Results from an RCT may not be published for 5–6 years, by which time the iPhone operating system has undergone substantial changes to functionality, design, and overall use. These operating system changes would result in the evaluated app feeling out of date at best and non-functional at worst. As such, the knowledge gained from that efficacy trial would be minimally useful for supporting current decisions about using that app. Other difficulties include the idiosyncratic wants and needs of users and the influence of context on effectiveness.

However, the public, patients, clinicians, policymakers, and healthcare commissioners have to make decisions on DHI now, and researchers need to support such decision making by creating an actionable knowledge base to identify the most effective, cost effective, safe, and scalable interventions (and components) for improving individual and population health. These decisions are particularly important in resource-constrained contexts.

This paper explores issues that arise in developing an accumulating knowledge base around DHIs, and how this knowledge can be generated in a timely manner, using scarce resources efficiently. The approach is pragmatic, with a focus on decision making and moving the science forward, generating cumulative knowledge around identifying important components, and working out how to test them with a view to improving the quality and effectiveness of DHIs and the efficiency of the research process. This paper is written from the perspective of a body charged with appraising evidence for using specific DHI within a publically funded, resource-limited health system, such as the United Kingdom National Institute for Health and Care Excellence.

This paper does not seek to provide detailed analysis of appropriate design features of evaluation studies, such as choice of comparators, outcome measures, mediator and moderator variables, study samples, or the occasions when particular study designs are a better fit with the evaluation context. These are important issues for which a literature is beginning to emerge.15, 16

Section snippets

Structure

The paper starts by defining the research questions (RQs) that, in the authors’ opinion, should form the basis for an appraisal of a DHI (Table 1). It then considers appropriate research methods for each of these RQs. Where the appropriate methods are largely similar to those used in research of other (non-digital) complex interventions, readers are referred to the appropriate references. Where there are novel or specific issues that arise, or are particularly salient, in evaluation of DHIs,

Defining the Problem

1. Is There a Clear Health Need That This Digital Health Intervention Is Intended to Address?

2. Is There a Defined Population That Could Benefit From This Digital Health Intervention?

As with any complex intervention, consideration of the likely benefits of a DHI starts with a detailed and preferably theory-based characterization of the nature of the problem and the context in which the intervention will be used.22, 23, 24

Defining the Likely Benefit of the Digital Health Intervention

3. Is the Digital Health Intervention Likely to Reach This Population, and if so, Is the Population Likely to Use it?

The concepts of reach, uptake, and context are particularly salient for DHIs, as impact and cost effectiveness are highly dependent on the total number of users,25 and effectiveness may be highly dependent on context. For example, effects seen when a DHI is used in a controlled environment (laboratory or clinical office) may not be replicated if used in the “wild,” with many

Discussion and Conclusions

This paper outlines an RQ-driven approach to the evaluation of DHI, which should lead to an accumulating knowledge base around such interventions in a timely and resource-efficient manner. Good research in this area requires fertile multidisciplinary collaborations that draw on insights and experience from multiple fields, including clinical medicine, health services research, behavioral science, education, engineering, and computer science. Researchers from an engineering or computer science

Acknowledgments

This 2016 theme issue of the American Journal of Preventive Medicine is supported by funding from the NIH Office of Behavioral and Social Sciences Research (OBSSR) to support the dissemination of research on digital health interventions, methods, and implications for preventive medicine.

This paper is one of the outputs of two workshops, one supported by the Medical Research Council (MRC)/National Institute for Health Research (NIHR) Methodology Research Program (PI Susan Michie), the OBSSR

References (61)

  • Z. Khadjesari et al.

    Can stand-alone computer-based interventions reduce alcohol consumption? A systematic review

    Addiction

    (2011)
  • E. Murray et al.

    Interactive Health Communication Applications for people with chronic disease

    Cochrane Database Syst Rev

    (2005)
  • G. McLean et al.

    Digital interventions to promote self-management in adults with hypertension systematic review and meta-analysis

    J Hypertens

    (2016)
  • K. Pal et al.

    Computer-based diabetes self-management interventions for adults with type 2 diabetes mellitus

    Cochrane Database Syst Rev

    (2013)
  • C. Hollis et al.

    Technological innovations in mental healthcare: harnessing the digital revolution

    Br J Psychiatry

    (2015)
  • F.K. Arnberg et al.

    Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness

    PLoS One

    (2014)
  • J.V. Olthuis et al.

    Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults

    Cochrane Database Syst Rev

    (2015)
  • A. McCombie et al.

    Computerised cognitive behavioural therapy for psychological distress in patients with physical illnesses: a systematic review

    J Clin Psychol Med Settings

    (2015)
  • S. van Beugen et al.

    Internet-based cognitive behavioral therapy for patients with chronic somatic conditions: a meta-analytic review

    J Med Internet Res

    (2014)
  • K. Patrick et al.

    Rapid rate of technological development and its implications for research on digital behavior interventions

    Am J Prev Med

    (2016)
  • E. Murray et al.

    Methodological challenges in online trials

    J Med Internet Res

    (2009)
  • Michie S, West R. Guide to Development and Evaluation of Digital Behaviour Change Interventions in Healthcare. London:...
  • N. Titov et al.

    MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression

    Psychiatr Serv

    (2015)
  • P. Bower et al.

    Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data

    BMJ

    (2013)
  • F.L. Hamilton et al.

    DIgital Alcohol Management ON Demand (DIAMOND) feasibility randomised controlled trial of a web-based intervention to reduce alcohol consumption in people with hazardous and harmful use versus a face-to-face intervention: protocol

    Pilot Feasibility Stud

    (2015)
  • L. Dennison et al.

    Does brief telephone support improve engagement with a web-based weight management intervention? Randomized controlled trial

    J Med Internet Res

    (2014)
  • N. Titov et al.

    Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance

    PLoS ONE

    (2010)
  • P. Craig et al.

    Developing and evaluating complex interventions: the new Medical Research Council guidance

    BMJ

    (2008)
  • N.C. Campbell et al.

    Designing and evaluating complex interventions to improve health care

    BMJ

    (2007)
  • E. Murray et al.

    Normalisation process theory: a framework for developing, evaluating and implementing complex interventions

    BMC Med

    (2010)
  • Cited by (403)

    View all citing articles on Scopus

    This article is part of a theme section titled Digital Health: Leveraging New Technologies to Develop, Deploy, and Evaluate Behavior Change Interventions.

    View full text