A framework for evaluating eHealth research

https://doi.org/10.1016/j.evalprogplan.2006.08.009Get rights and content

Abstract

Health care is in the midst of a consumer-oriented technology explosion. Individuals of all ages and backgrounds have discovered eHealth. But the challenges of implementing and evaluating eHealth are just beginning to surface, and, as technology changes, new challenges emerge. Evaluation is critical to the future of eHealth. This article addresses four dimensions of eHealth evaluation: (1) design and methodology issues; (2) challenges related to the technology itself; (3) environmental issues that are not specific to eHealth but pose special problems for eHealth researchers; and (4) logistic or administrative concerns of the evaluation methodology selected. We suggest that these four dimensions must be integrated to provide a holistic framework for designing and implementing eHealth research projects, as well as for understanding the totality of the eHealth intervention. The framework must be flexible enough to adapt to a variety of end users, regardless of whether the end user is a healthcare organization, a for-profit business, a community organization, or an individual. The framework is depicted as a puzzle with four interlocking pieces.

Introduction

Health care is in the midst of a consumer-oriented technology explosion. Individuals of all ages and backgrounds have discovered eHealth. For example, we can now gain access to our own medical records through web portals, “chat” online with others individually or in virtual communities, transmit health data using the telephone or wireless technologies, and surf the Internet to find information about health and health services. But the challenges of implementing and evaluating eHealth are just beginning to surface, and, as technology changes, new challenges emerge. Evaluation is critical to the future of eHealth. Unfortunately, few blueprints for effective evaluation methodologies come with this 21st century technology. This article addresses four dimensions of eHealth evaluation: (1) design and methodology issues; (2) challenges related to the technology itself; (3) environmental issues that are not specific to eHealth but pose special problems for eHealth researchers; and (4) logistic or administrative concerns of the evaluation methodology selected.

This article describes challenges associated with each dimension, using examples from actual eHealth interventions; proposes that thorough planning for each dimension is “necessary but not sufficient” for sound eHealth research; and suggests that the four dimensions must be integrated to provide a holistic framework for designing and implementing eHealth research projects, as well as for understanding the totality of the eHealth intervention. The framework must be flexible enough to adapt to a variety of end users, regardless of whether the end user is a healthcare organization, a for-profit business, a community organization, or an individual. The framework is depicted as a puzzle with four interlocking pieces (Fig. 1).

This article does not endorse specific designs, methods, or approaches for conducting eHealth research; these decisions should be dependent on the question being asked, the nature of the technology, and the specific needs of the organization. Nor does it address sustainability issues; these topics are beyond the scope of this document and have been addressed in the literature (Maheu, Whitten, & Allen, 2001). Rather, the contribution of this paper is to explore the challenges and opportunities in eHealth research from a holistic perspective, in order to advance our understanding of consumer technologies that may have an impact on health status.

Section snippets

The eHealth explosion

In The Wizard of Oz, Dorothy dreams that her farmhouse is tossed in the vortex of a tornado and lands squarely on the Wicked Witch. With the help of the Scarecrow, the Tin Man, and the Lion, she skips along the yellow brick road in search of answers from the Great Oz. The Land of Oz is a good metaphor for our healthcare system today. Like Dorothy's farmhouse, our healthcare system has been uprooted by advances in telecommunications and electronics, and keeping pace with these advances can be

eHealth research challenges

The explosion in eHealth use has generated a corresponding increase in the literature describing these technologies. A call for papers to the British Medical Journal on eHealth issues resulted in nearly 100 submissions—more than any submitted for a BMJ theme issue (Jadad & Delamothe, 2004). But the editors noted that many of the submissions were descriptive in nature or single-case studies. They question whether traditional research methods are adequate, even appropriate, to investigate eHealth

A call for innovative eHealth research

eHealth offers a new lens through which to view the delivery of health services, including preventive and acute care. But with this opportunity comes a challenge to lay the groundwork for a meaningful research agenda. We have presented an overview of eHealth research issues, drawing on our perspectives from two very different projects. Although the patient populations, technologies, and approaches differ, they share common concerns that can provide valuable lessons to others who are interested

Acknowledgments

The authors acknowledge, with appreciation and thanks, the financial support provided by the Health e-Technologies Initiative, and the assistance provided by the National Program Staff of the Robert Wood Johnson Foundation.

References (15)

  • N.T. Artinian et al.

    Self-care behaviors among patients with heart failure

    Heart and Lung

    (2002)
  • T. Baranowski et al.

    The Fun, Food and Fitness Project (FFFP): The Baylor GEMS pilot study

    Ethnicity & Diseases

    (2003)
  • Dansky, K.H. (2005). Technology enablers: Management practices that promote end-user acceptance of technology in health...
  • K.H. Dansky et al.

    Clinical outcomes of telehomecare for diabetic patients

    Journal on Information Technology in Healthcare

    (2003)
  • T.R. Eng

    eHealth research and evaluation: Challenges and opportunities

    Journal of Health Communication

    (2002)
  • T. Ferguson et al.

    The first generation of e-patients

    British Medical Journal

    (2004)
  • R.E. Hurley

    Qualitative research and the profound grasp of the obvious

    Health Services Research

    (1998)
There are more references available in the full text version of this article.

Cited by (56)

  • Revealing the role of intellectual capital in digitalized health networks. A meso‑level analysis for building and monitoring a KPI dashboard

    2022, Technological Forecasting and Social Change
    Citation Excerpt :

    On the other hand, Pagliari (2007) stated that a multidisciplinary team of experts is irreplaceable in order to capture the potential of digital innovation. Dansky et al. (2006) argued that multidisciplinary elements and skills are aimed at developing, organizing, monitoring, and reshaping the digital healthcare system. Hamid and Skarmad (2008) shed light on the necessity to adopt a multiple perspective in the evaluation phase.

  • Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

    2017, Evaluation and Program Planning
    Citation Excerpt :

    For instance, logistical delays, as well as the need for additional training in using the technological equipment, can delay program start and may need to be attended to in order to secure quality data collection. For a good review on the topic, see Dansky, Thompson, and Sanner (2006), who outlined four main challenges to engaging in TMH research, many examples of which were relevant in the case of the current example: 1) design and methodology issues; 2) challenges related to the technology; 3) environmental issues not specific TMH but nonetheless relevant; and 4) logistical and administrative issues. Evaluators will need to be increasingly familiar with these challenges and potential methods to overcome such barriers when engaging in program evaluation efforts incorporating the use of technology.

  • Principles for the evaluation of telemedicine applications: Results of a systematic review and consensus process

    2016, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
View all citing articles on Scopus
View full text