Evaluating Digital Health Interventions: Key Questions and Approaches
Introduction
There is enormous potential for digital health interventions (DHIs; i.e., interventions delivered via digital technologies such as smartphones, website, or text messaging) to provide effective, cost effective, safe, and scalable interventions to improve health and healthcare. DHIs can be used to promote healthy behaviors (e.g., smoking cessation,1 healthy eating,2 physical activity,3 safer sex,4 or alcohol consumption5); improve outcomes in people with long-term conditions6 such as cardiovascular disease,7 diabetes,8 and mental health conditions9; and provide remote access to effective treatments (e.g., computerized cognitive behavioral therapy for mental health and somatic problems).10, 11, 12, 13 They are typically complex interventions with multiple components, and many have multiple aims, including enabling users to be better informed about their health, share experiences with others in similar positions, change perceptions and cognitions around health, assess and monitor specified health states or health behaviors, titrate medication, clarify health priorities and reach treatment decisions congruent with these, and improve communication between patients and healthcare professionals (HCPs). Active components may include information, psycho-education, personal stories, formal decision aids, behavior change support, interactions with HCPs and other patients, self-assessment or monitoring tools (questionnaires, wearables, monitors), and effective theory-based psychological interventions developed for face-to-face delivery, such as cognitive behavioral therapy or mindfulness training.
To date, the potential of DHIs has scarcely been realized, partly because of difficulties in generating an accumulating knowledge base for guiding decisions about DHIs. These include the rapid change of the wider technology landscape,14 which requires DHIs to constantly evolve and be updated just to remain useful, let alone improve. For example, imagine an iPhone app promoting physical activity, with development and evaluation starting in 2008. Results from an RCT may not be published for 5–6 years, by which time the iPhone operating system has undergone substantial changes to functionality, design, and overall use. These operating system changes would result in the evaluated app feeling out of date at best and non-functional at worst. As such, the knowledge gained from that efficacy trial would be minimally useful for supporting current decisions about using that app. Other difficulties include the idiosyncratic wants and needs of users and the influence of context on effectiveness.
However, the public, patients, clinicians, policymakers, and healthcare commissioners have to make decisions on DHI now, and researchers need to support such decision making by creating an actionable knowledge base to identify the most effective, cost effective, safe, and scalable interventions (and components) for improving individual and population health. These decisions are particularly important in resource-constrained contexts.
This paper explores issues that arise in developing an accumulating knowledge base around DHIs, and how this knowledge can be generated in a timely manner, using scarce resources efficiently. The approach is pragmatic, with a focus on decision making and moving the science forward, generating cumulative knowledge around identifying important components, and working out how to test them with a view to improving the quality and effectiveness of DHIs and the efficiency of the research process. This paper is written from the perspective of a body charged with appraising evidence for using specific DHI within a publically funded, resource-limited health system, such as the United Kingdom National Institute for Health and Care Excellence.
This paper does not seek to provide detailed analysis of appropriate design features of evaluation studies, such as choice of comparators, outcome measures, mediator and moderator variables, study samples, or the occasions when particular study designs are a better fit with the evaluation context. These are important issues for which a literature is beginning to emerge.15, 16
Section snippets
Structure
The paper starts by defining the research questions (RQs) that, in the authors’ opinion, should form the basis for an appraisal of a DHI (Table 1). It then considers appropriate research methods for each of these RQs. Where the appropriate methods are largely similar to those used in research of other (non-digital) complex interventions, readers are referred to the appropriate references. Where there are novel or specific issues that arise, or are particularly salient, in evaluation of DHIs,
Defining the Problem
1. Is There a Clear Health Need That This Digital Health Intervention Is Intended to Address?
2. Is There a Defined Population That Could Benefit From This Digital Health Intervention?
As with any complex intervention, consideration of the likely benefits of a DHI starts with a detailed and preferably theory-based characterization of the nature of the problem and the context in which the intervention will be used.22, 23, 24
Defining the Likely Benefit of the Digital Health Intervention
3. Is the Digital Health Intervention Likely to Reach This Population, and if so, Is the Population Likely to Use it?
The concepts of reach, uptake, and context are particularly salient for DHIs, as impact and cost effectiveness are highly dependent on the total number of users,25 and effectiveness may be highly dependent on context. For example, effects seen when a DHI is used in a controlled environment (laboratory or clinical office) may not be replicated if used in the “wild,” with many
Discussion and Conclusions
This paper outlines an RQ-driven approach to the evaluation of DHI, which should lead to an accumulating knowledge base around such interventions in a timely and resource-efficient manner. Good research in this area requires fertile multidisciplinary collaborations that draw on insights and experience from multiple fields, including clinical medicine, health services research, behavioral science, education, engineering, and computer science. Researchers from an engineering or computer science
Acknowledgments
This 2016 theme issue of the American Journal of Preventive Medicine is supported by funding from the NIH Office of Behavioral and Social Sciences Research (OBSSR) to support the dissemination of research on digital health interventions, methods, and implications for preventive medicine.
This paper is one of the outputs of two workshops, one supported by the Medical Research Council (MRC)/National Institute for Health Research (NIHR) Methodology Research Program (PI Susan Michie), the OBSSR
References (61)
- et al.
Smoking cessation support delivered via mobile phone text messaging (txt2stop): a single-blind, randomised trial
Lancet
(2011) - et al.
Website-delivered physical activity interventions a review of the literature
Am J Prev Med
(2007) Methods to support human-centred design
Int J Hum Comput Stud
(2001)- et al.
How we design feasibility studies
Am J Prev Med
(2009) - et al.
Web-based smoking-cessation programs: results of a randomized trial
Am J Prev Med
(2008) - et al.
Using engineering control principles to inform the design of adaptive interventions: a conceptual introduction
Drug Alcohol Depend
(2007) - et al.
Factorial experiments: efficient tools for evaluation of intervention components
Am J Prev Med
(2014) External validity of randomised controlled trials: “to whom do the results of this trial apply?”
Lancet
(2005)- et al.
Adaptive e-learning to improve dietary behaviour: a systematic review and cost-effectiveness analysis
Health Technol Assess
(2011) - et al.
Interactive computer-based interventions for sexual health promotion
Cochrane Database Syst Rev
(2010)
Can stand-alone computer-based interventions reduce alcohol consumption? A systematic review
Addiction
Interactive Health Communication Applications for people with chronic disease
Cochrane Database Syst Rev
Digital interventions to promote self-management in adults with hypertension systematic review and meta-analysis
J Hypertens
Computer-based diabetes self-management interventions for adults with type 2 diabetes mellitus
Cochrane Database Syst Rev
Technological innovations in mental healthcare: harnessing the digital revolution
Br J Psychiatry
Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness
PLoS One
Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults
Cochrane Database Syst Rev
Computerised cognitive behavioural therapy for psychological distress in patients with physical illnesses: a systematic review
J Clin Psychol Med Settings
Internet-based cognitive behavioral therapy for patients with chronic somatic conditions: a meta-analytic review
J Med Internet Res
Rapid rate of technological development and its implications for research on digital behavior interventions
Am J Prev Med
Methodological challenges in online trials
J Med Internet Res
MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression
Psychiatr Serv
Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data
BMJ
DIgital Alcohol Management ON Demand (DIAMOND) feasibility randomised controlled trial of a web-based intervention to reduce alcohol consumption in people with hazardous and harmful use versus a face-to-face intervention: protocol
Pilot Feasibility Stud
Does brief telephone support improve engagement with a web-based weight management intervention? Randomized controlled trial
J Med Internet Res
Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance
PLoS ONE
Developing and evaluating complex interventions: the new Medical Research Council guidance
BMJ
Designing and evaluating complex interventions to improve health care
BMJ
Normalisation process theory: a framework for developing, evaluating and implementing complex interventions
BMC Med
Cited by (403)
Promises and Perils of Consumer Mobile Technologies in Cardiovascular Care: JACC Scientific Statement
2024, Journal of the American College of CardiologyVirtual physical therapy is a huge step forward for the profession
2024, KinesitherapieStandUPTV: Preparation and optimization phases of a mHealth intervention to reduce sedentary screen time in adults
2024, Contemporary Clinical TrialsHome-based exercise interventions delivered by technology in older adults: A scoping review of technological tools usage
2024, International Journal of Medical InformaticsPersonalization in digital health interventions for older adults with cancer: A scoping review
2023, Journal of Geriatric Oncology
This article is part of a theme section titled Digital Health: Leveraging New Technologies to Develop, Deploy, and Evaluate Behavior Change Interventions.