Skip to main content

Advertisement

SpringerLink
Log in

A Review of Usability Evaluation Methods and Their Use for Testing eHealth HIV Interventions

  • eHealth and HIV (J Simoni and B Guthrie, Section Editors)
  • Published:
Current HIV/AIDS Reports Aims and scope Submit manuscript

Abstract

Purpose of Review

To provide a comprehensive review of usability testing of eHealth interventions for HIV.

Recent Findings

We identified 28 articles that assessed the usability of eHealth interventions for HIV, most of which were published within the past 3 years. The majority of the eHealth interventions for HIV was developed on a mobile platform and focused on HIV prevention as the intended health outcome. Usability evaluation methods included eye-tracking, questionnaires, semi-structured interviews, contextual interviews, think-aloud protocols, cognitive walkthroughs, heuristic evaluations and expert reviews, focus groups, and scenarios.

Summary

A wide variety of methods is available to evaluate the usability of eHealth interventions. Employing multiple methods may provide a more comprehensive assessment of the usability of eHealth interventions as compared with inclusion of only a single evaluation method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance

  1. Hollander R. Two-thirds of the world’s population are now connected by mobile devices. Bus Insid. 2017.

  2. Holst A. Number of smartphone users worldwide 2014–2020. Statista, Available at: https://www.statista.com/statistics/330695/number …; 2019.

  3. Center PR. Communications technology in emerging and developing nations. Washington, D.C.: Pew Research Center; 2015.

    Google Scholar 

  4. Akhlaq A, McKinstry B, Muhammad KB, Sheikh A. Barriers and facilitators to health information exchange in low-and middle-income country settings: a systematic review. Health Policy Plan. 2016;31(9):1310–25.

    Article  Google Scholar 

  5. Eysenbach G. What is e-health? J Med Internet Res. 2001;3(2):e20. https://doi.org/10.2196/jmir.3.2.e20.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Page TF, Horvath KJ, Danilenko GP, Williams M. A cost analysis of an internet based medication adherence intervention for people living with HIV. J Acquir Immune Defic Ssyndr (1999). 2012;60(1):1.

    Article  Google Scholar 

  7. Patel AR, Kessler J, Braithwaite RS, Nucifora KA, Thirumurthy H, Zhou Q, et al. Economic evaluation of mobile phone text message interventions to improve adherence to HIV therapy in Kenya. Medicine. 2017;96(7).

    Article  CAS  Google Scholar 

  8. Schnall R, Travers J, Rojas M, Carballo-Diéguez A. eHealth interventions for HIV prevention in high-risk men who have sex with men: a systematic review. J Med Internet Res. 2014;16(5):e134.

    Article  Google Scholar 

  9. Noar SM, Willoughby JF. eHealth interventions for HIV prevention. AIDS Care. 2012;24(8):945–52.

    Article  Google Scholar 

  10. Chaiyachati KH, Ogbuoji O, Price M, Suthar AB, Negussie EK, Bärnighausen T. Interventions to improve adherence to antiretroviral therapy: a rapid systematic review. Aids. 2014;28:S187–204.

    Article  Google Scholar 

  11. Henny KD, Wilkes AL, McDonald CM, Denson DJ, Neumann MS. A rapid review of eHealth interventions addressing the continuum of HIV care (2007–2017). AIDS Behav. 2018;22(1):43–63.

    Article  Google Scholar 

  12. Bevan N, Carter J, Earthy J, Geis T, Harker S, editors. New ISO standards for usability, usability reports and usability measures. Human-computer interaction. Theory, design, development and practice. Cham: Springer International Publishing; 2016.

  13. Seffah A, Kececi N, Donyaee M, editors. QUIM: a framework for quantifying usability metrics in software quality models. Proceedings Second Asia-Pacific Conference on Quality Software. IEEE; 2001.

  14. Sheehan B, Lee Y, Rodriguez M, Tiase V, Schnall R. A comparison of usability factors of four mobile devices for accessing healthcare information by adolescents. Appl Clin Inform. 2012;3(04):356–66.

    Article  CAS  Google Scholar 

  15. Yen P-Y, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc. 2012;19(3):413–22.

    Article  Google Scholar 

  16. Kaufman D, Roberts WD, Merrill J, Lai T-Y, Bakken S. Applying an evaluation framework for health information system design, development, and implementation. Nurs Res. 2006;55(2):S37–42.

    Article  Google Scholar 

  17. • Beauchemin M, Gradilla M, Baik D, Cho H, Schnall R. A multi-step usability evaluation of a self-management app to support medication adherence in persons living with HIV. Int J Med Inform. 2019;122:37–44. https://doi.org/10.1016/j.ijmedinf.2018.11.012. This article describes the use of a cognitive walkthrough for evaluating the usability of an HIV prevention intervention. Cognitive walkthrough is a rarely used evaluation method but is very useful for pinpointing challenging tasks or complicated features associated with an eHealth intervention.

    Article  PubMed  Google Scholar 

  18. Cho H, Porras T, Baik D, Beauchemin M, Schnall R. Understanding the predisposing, enabling, and reinforcing factors influencing the use of a mobile-based HIV management app: a real-world usability evaluation. Int J Med Inform. 2018;117:88–95.

    Article  Google Scholar 

  19. Cho H, Powell D, Pichon A, Thai J, Bruce J, Kuhns LM, et al. A mobile health intervention for HIV prevention among racially and ethnically diverse young men: usability evaluation. JMIR mHealth uHealth. 2018;6(9):e11450.

    Article  Google Scholar 

  20. • Cho H, Yen P-Y, Dowding D, Merrill JA, Schnall R. A multi-level usability evaluation of mobile health applications: a case study. J Biomed Inform. 2018;86:79–89. This paper provides an overview of the combination of multiple usability methods to develop a self-management app.

    Article  Google Scholar 

  21. Cho H, Powell D, Pichon A, Kuhns LM, Garofalo R, Schnall R. Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation. Int J Med Inform. 2019;129:366–73. https://doi.org/10.1016/j.ijmedinf.2019.07.010.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Coppock D, Zambo D, Moyo D, Tanthuma G, Chapman J, Re VL III, et al. Development and usability of a smartphone application for tracking antiretroviral medication refill data for human immunodeficiency virus. Methods Inf Med. 2017;56(05):351–9.

    Article  Google Scholar 

  23. Cordova D, Alers-Rojas F, Lua FM, Bauermeister J, Nurenberg R, Ovadje L, et al. The usability and acceptability of an adolescent mHealth HIV/STI and drug abuse preventive intervention in primary care. Behav Med. 2018;44(1):36–47.

    Article  Google Scholar 

  24. Danielson CK, McCauley JL, Gros KS, Jones AM, Barr SC, Borkman AL, et al. SiHLE Web. com: development and usability testing of an evidence-based HIV prevention website for female African-American adolescents. Health Inf J. 2016;22(2):194–208.

    Article  Google Scholar 

  25. Hightow-Weidman LB, Fowler B, Kibe J, McCoy R, Pike E, Calabria M, et al. HealthMpowerment.org: development of a theory-based HIV/STI website for young black MSM. AIDS Educ Prev. 2011;23(1):1–12. https://doi.org/10.1521/aeap.2011.23.1.1.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Himelhoch S, Kreyenbuhl J, Palmer-Bacon J, Chu M, Brown C, Potts W. Pilot feasibility study of Heart2HAART: a smartphone application to assist with adherence among substance users living with HIV. AIDS Care. 2017;29(7):898–904. https://doi.org/10.1080/09540121.2016.1259454.

    Article  PubMed  Google Scholar 

  27. Horvath KJ, Bauermeister JA. eHealth literacy and intervention tailoring impacts the acceptability of a HIV/STI testing intervention and sexual decision making among young gay and bisexual men. AIDS Educ Prev. 2017;29(1):14–23. https://doi.org/10.1521/aeap.2017.29.1.14.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Kawakyu N, Nduati R, Munguambe K, Coutinho J, Mburu N, DeCastro G, et al. Development and implementation of a mobile phone-based prevention of mother-to-child transmission of HIV cascade analysis tool: usability and feasibility testing in Kenya and Mozambique. JMIR mHealth and uHealth. 2019;7(5):e13963-e. https://doi.org/10.2196/13963.

    Article  Google Scholar 

  29. Luque AE, Corales R, Fowler RJ, DiMarco J, van Keken A, Winters P, et al. Bridging the digital divide in HIV care: a pilot study of an iPod personal health record. J Int Assoc Provid AIDS Care. 2013;12(2):117–21. https://doi.org/10.1177/1545109712457712.

    Article  PubMed  Google Scholar 

  30. Mitchell JT, LeGrand S, Hightow-Weidman LB, McKellar MS, Kashuba AD, Cottrell M, et al. Smartphone-based contingency management intervention to improve pre-exposure prophylaxis adherence: pilot trial. JMIR mHealth uHealth. 2018;6(9):e10456-e. https://doi.org/10.2196/10456.

    Article  Google Scholar 

  31. Musiimenta A, Atukunda EC, Tumuhimbise W, Pisarski EE, Tam M, Wyatt MA, et al. Acceptability and feasibility of real-time antiretroviral therapy adherence interventions in rural Uganda: mixed-method pilot randomized controlled trial. JMIR mHealth and uHealth. 2018;6(5):e122-e. https://doi.org/10.2196/mhealth.9031.

    Article  Google Scholar 

  32. Sabben G, Mudhune V, Ondeng’e K, Odero I, Ndivo R, Akelo V, et al. A smartphone game to prevent HIV among young Africans (Tumaini): assessing intervention and study acceptability among adolescents and their parents in a randomized controlled trial. JMIR mHealth and uHealth. 2019;7(5):e13049-e. https://doi.org/10.2196/13049.

    Article  Google Scholar 

  33. Schnall R, Bakken S, Brown Iii W, Carballo-Dieguez A, Iribarren S. Usabilty evaluation of a prototype mobile app for health management for persons living with HIV. Stud Health Technol Inform. 2016;225:481–5.

    PubMed  PubMed Central  Google Scholar 

  34. Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform. 2016;60:243–51. https://doi.org/10.1016/j.jbi.2016.02.002.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Shegog R, Markham CM, Leonard AD, Bui TC, Paul ME. “+CLICK”: pilot of a web-based training program to enhance ART adherence among HIV-positive youth. AIDS Care. 2012;24(3):310–8. https://doi.org/10.1080/09540121.2011.608788.

    Article  PubMed  Google Scholar 

  36. Shegog R, Markham C, Peskin M, Dancel M, Coton C, Tortolero S, editors. ‘It’s your game’: an innovative multimedia virtual world to prevent HIV/STI and pregnancy in middle school youth. Medinfo 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics; Building Sustainable Health Systems. Amsterdam: IOS Press; 2007.

  37. Skeels MM, Kurth A, Clausen M, Severynen A, Garcia-Smith H, editors. CARE+ user study: usability and attitudes towards a tablet pc computer counseling tool for HIV+ men and women. AMIA Annual Symposium Proceedings. American Medical Informatics Association; 2006.

  38. Stonbraker S, Cho H, Hermosi G, Pichon A, Schnall R. Usability testing of a mhealth app to support self-management of HIV-associated non-AIDS related symptoms. Stud Health Technol Inform. 2018;250:106.

    PubMed  PubMed Central  Google Scholar 

  39. Sullivan PS, Driggers R, Stekler JD, Siegler A, Goldenberg T, McDougal SJ, et al. Usability and acceptability of a mobile comprehensive HIV prevention app for men who have sex with men: a pilot study. JMIR mHealth and uHealth. 2017;5(3):e26.

    Article  Google Scholar 

  40. Rodríguez Vargas B, Sánchez-Rubio Ferrández J, Garrido Fuentes J, Velayos R, Morillo Verdugo R, Sala Piñol F, et al. Usability and acceptability of a comprehensive HIV and other sexually transmitted infections prevention app. J Med Syst. 2019;43(6):175. https://doi.org/10.1007/s10916-019-1323-4.

    Article  PubMed  Google Scholar 

  41. Widman L, Golin CE, Kamke K, Massey J, Prinstein MJ. Feasibility and acceptability of a web-based HIV/STD prevention program for adolescent girls targeting sexual communication skills. Health Educ Res. 2017;32(4):343–52. https://doi.org/10.1093/her/cyx048.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Williams M, Bowen A, Ei S. An evaluation of the experiences of rural MSM who accessed an online HIV/AIDS health promotion intervention. Health Promot Pract. 2010;11(4):474–82. https://doi.org/10.1177/1524839908324783.

    Article  PubMed  Google Scholar 

  43. Winstead-Derlega C, Rafaly M, Delgado S, Freeman J, Cutitta K, Miles T, et al. A pilot study of delivering peer health messages in an HIV clinic via mobile media. Telemed e-Health. 2012;18(6):464–9.

    Article  Google Scholar 

  44. Ybarra M, Biringi R, Prescott T, Bull SS. Usability and navigability of an HIV/AIDS internet intervention for adolescents in a resource limited setting. Comput Inform Nurs. 2012;30(11):587.

    Article  Google Scholar 

  45. US Department of Health Human Services. Usability testing. www.usability.gov/how-to-and-tools/methods/usability-testing.html. Accessed 2017-12-29. [WebCite Cache ID 6w4kyZEBQ], 2012.

  46. Gruenstein A, McGraw I, Badr I, editors. The WAMI toolkit for developing, deploying, and evaluating web-accessible multimodal interfaces. Proceedings of the 10th international conference on Multimodal interfaces; 2008.

  47. Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ (Clin Res Ed). 1995;311(6996):42–5. https://doi.org/10.1136/bmj.311.6996.42.

    Article  CAS  Google Scholar 

  48. DiCicco-Bloom B, Crabtree BF. The qualitative research interview. Med Educ. 2006;40(4):314–21. https://doi.org/10.1111/j.1365-2929.2006.02418.x.

    Article  PubMed  Google Scholar 

  49. Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform. 2009;78(5):340–53.

    Article  Google Scholar 

  50. Kaufman DR, Patel VL, Hilliman C, Morin PC, Pevzner J, Weinstock RS, et al. Usability in the real world: assessing medical information technologies in patients’ homes. J Biomed Inform. 2003;36(1–2):45–60.

    Article  Google Scholar 

  51. Lewis JR, Sauro J, editors. The factor structure of the system usability scale. International conference on human centered design. Berlin: Springer; 2009.

    Google Scholar 

  52. •• Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for usability assessment of mobile health technology: validation study. JMIR mHealth and uHealth. 2018;6(1):e4. This study validates the use of a customizable usability questionnaire for assessing the usability of mobile technology.

    Article  Google Scholar 

  53. Lewis JR. Psychometric evaluation of the PSSUQ using data from five years of usability studies. Int J Hum Comput Interact. 2002;14(3–4):463–88.

    Article  Google Scholar 

  54. Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact. 1995;7(1):57–78.

    Article  Google Scholar 

  55. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694–6.

    Article  CAS  Google Scholar 

Download references

Acknowledgments

We would like to acknowledge the contributions of John Usseglio, an Informationist at Columbia University Irving Medical Center. Mr. Usseglio provided his expertise on constructing comprehensive search strategies for this review.

Funding

At the time of submission of this manuscript, RD was funded by the Mervyn W. Susser Post-doctoral Fellowship Program at the Gertrude H. Sergievsky Center. RS was supported by the National Institute of Nursing Research of the National Institutes of Health under award number K24NR018621. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Authors and Affiliations

  1. Gertrude H. Sergievsky Center, College of Physicians and Surgeons, Columbia University Medical Center, 630 W 168th Street, New York, NY, 10032, USA

    Rindcy Davis

  2. Department of Epidemiology, Mailman School of Public Health, Columbia University Medical Center, 630 W 168th Street, New York, NY, 10032, USA

    Jessica Gardner

  3. School of Nursing, Columbia University, New York, NY, 10032, USA

    Rebecca Schnall

Authors
  1. Rindcy Davis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jessica Gardner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rebecca Schnall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rebecca Schnall.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Human and Animal Rights

All reported studies/experiments with human or animal subjects performed by authors have been previously published and complied with all applicable ethical standards (including Helsinki declaration and its amendments, institutional/national research committee standards, and international/national/institutional guidelines).

Disclaimer

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on eHealth and HIV

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Davis, R., Gardner, J. & Schnall, R. A Review of Usability Evaluation Methods and Their Use for Testing eHealth HIV Interventions. Curr HIV/AIDS Rep 17, 203–218 (2020). https://doi.org/10.1007/s11904-020-00493-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11904-020-00493-3

Keywords

Search

Navigation