Towards Usable E-Health

 

 Permissions and Reprints

Summary

Background: The use of e-health can lead to several positive outcomes. However, the potential for e-health to improve healthcare is partially dependent on its ease of use. In order to determine the usability for any technology, rigorously developed and appropriate measures must be chosen.

Objectives: To identify psychometrically tested questionnaires that measure usability of e-health tools, and to appraise their generalizability, attributes coverage, and quality.

Methods: We conducted a systematic review of studies that measured usability of e-health tools using four databases (Scopus, PubMed, CINAHL, and HAPI). Non-primary research, studies that did not report measures, studies with children or people with cognitive limitations, and studies about assistive devices or medical equipment were systematically excluded. Two authors independently extracted information including: questionnaire name, number of questions, scoring method, item generation, and psychometrics using a data extraction tool with pre-established categories and a quality appraisal scoring table.

Results: Using a broad search strategy, 5,558 potentially relevant papers were identified. After removing duplicates and applying exclusion criteria, 35 articles remained that used 15 unique questionnaires. From the 15 questionnaires, only 5 were general enough to be used across studies. Usability attributes covered by the questionnaires were: learnability (15), efficiency (12), and satisfaction (11). Memorability (1) was the least covered attribute. Quality appraisal showed that face/content (14) and construct (7) validity were the most frequent types of validity assessed. All questionnaires reported reliability measurement. Some questionnaires scored low in the quality appraisal for the following reasons: limited validity testing (7), small sample size (3), no reporting of user centeredness (9) or feasibility estimates of time, effort, and expense (7).

Conclusions: Existing questionnaires provide a foundation for research on e-health usability. However, future research is needed to broaden the coverage of the usability attributes and psychometric properties of the available questionnaires.

Citation: Sousa VEC, Lopez KD. Towards usable e-health: A systematic review of usability questionnaires. Appl Clin Inform 2017; 8: 470–490 https://doi.org/10.4338/ACI-2016-10-R-0170

Human Subjects Protections

Human subjects were not included in the project.

• References

• 1 US Department of Health and Human Services. Expanding the reach and impact of consumer e-health tools. Washington, DC: : US Department of Health and Human Services; 2006. Available from: https://health.gov/communication/ehealth/ehealthtools/default.htm
  Google Scholar
• 2 Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci Transl Med 2015; 7 (283) 283rv3.
  CrossrefPubMedGoogle Scholar
• 3 Chen J, Bauman A, Allman-Farinelli M. A Study to Determine the Most Popular Lifestyle Smartphone Applications and Willingness of the Public to Share Their Personal Data for Health Research. Telemed J E Health 2016; 22 (08) 655-665.
  CrossrefPubMedGoogle Scholar
• 4 West JH, Hall PC, Hanson CL, Barnes MD, Giraud-Carrier C, Barrett J. There‘s an app for that: content analysis of paid health and fitness apps. J Med Internet Res 2012; 14 (03) e72.
  CrossrefPubMedGoogle Scholar
• 5 Silva BM, Rodrigues JJ, de la Torre Diez I, Lopez-Coronado M, Saleem K. Mobile-health: A review of current state in 2015. J Biomed Inform 2015; 56: 265-272.
  CrossrefPubMedGoogle Scholar
• 6 Benhamou PY, Melki V, Boizel R, Perreal F, Quesada JL, Bessieres-Lacombe S, Bosson JL, Halimi S, Hanaire H. One-year efficacy and safety of Web-based follow-up using cellular phone in type 1 diabetic patients under insulin pump therapy: the PumpNet study. Diabetes Metab 2007; 33 (03) 220-226.
  CrossrefPubMedGoogle Scholar
• 7 Ostojic V, Cvoriscec B, Ostojic SB, Reznikoff D, Stipic-Markovic A, Tudjman Z. Improving asthma control through telemedicine: a study of short-message service. Telemed J E Health 2005; 11 (01) 28-35.
  CrossrefPubMedGoogle Scholar
• 8 DeVito Dabbs A, Dew MA, Myers B, Begey A, Hawkins R, Ren D, Dunbar-Jacob J, Oconnell E, McCurry KR. Evaluation of a hand-held, computer-based intervention to promote early self-care behaviors after lung transplant. Clin Transplant 2009; 23 (04) 537-545.
  CrossrefPubMedGoogle Scholar
• 9 Vidrine DJ, Marks RM, Arduino RC, Gritz ER. Efficacy of cell phone-delivered smoking cessation counseling for persons living with HIV/AIDS: 3-month outcomes. Nicotine Tob Res 2012; 14 (01) 106-110.
  CrossrefPubMedGoogle Scholar
• 10 Lim MS, Hocking JS, Aitken CK, Fairley CK, Jordan L, Lewis JA, Hellard ME. Impact of text and email messaging on the sexual health of young people: a randomised controlled trial. J Epidemiol Community Health 2012; 66 (01) 69-74.
  CrossrefPubMedGoogle Scholar
• 11 Kunawararak P, Pongpanich S, Chantawong S, Pokaew P, Traisathit P, Srithanaviboonchai K, Plipat T. Tuberculosis treatment with mobile-phone medication reminders in northern Thailand. Southeast Asian J Trop Med Public Health 2011; 42 (06) 1444-1451.
  PubMedGoogle Scholar
• 12 Lester RT, Ritvo P, Mills EJ, Kariri A, Karanja S, Chung MH, Jack W, Habyarimana J, Sadatsafavi M, Najafzadeh M, Marra CA, Estambale B, Ngugi E, Ball TB, Thabane L, Gelmon LJ, Kimani J, Ackers M, Plummer FA. Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet 2010; 376 9755 1838-1845.
  CrossrefPubMedGoogle Scholar
• 13 Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, Patel V, Haines A. The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 2013; 10 (01) e1001362.
  CrossrefPubMedGoogle Scholar
• 14 Seidling HM, Stutzle M, Hoppe-Tichy T, Allenet B, Bedouch P, Bonnabry P, Coleman JJ, Fernandez-Llimos F, Lovis C, Rei MJ, Storzinger D, Taylor LA, Pontefract SK, van den Bemt PM, van der Sijs H, Haefeli WE. Best practice strategies to safeguard drug prescribing and drug administration: an anthology of expert views and opinions. Int J Clin Pharm 2016; 38 (02) 362-373.
  CrossrefPubMedGoogle Scholar
• 15 Kampmeijer R, Pavlova M, Tambor M, Golinowska S, Groot W. The use of e-health and m-health tools in health promotion and primary prevention among older adults: a systematic literature review. BMC Health Services Research 2016; 16 (05) 290.
  CrossrefPubMedGoogle Scholar
• 16 Abran A, Khelifi A, Suryn W, Seffah A. Usability meanings and interpretations in ISO standards. Software Qual J 2003; 11 (04) 325-338.
  CrossrefPubMedGoogle Scholar
• 17 International Organization for Standardization/International Electrotechnical Commission. ISO/IEC 9241–14 Ergonomic requirements for office work with visual display terminals (VDT)s –Part 14 Menu dialogues: ISO/IEC 9241–14; 1998
  PubMedGoogle Scholar
• 18 Nielsen J. Usability Engineering. Boston: Academic Press; 1993
  Google Scholar
• 19 Nielsen J. Lifetime Practice Award. Paris, France: SIGCHI; 2013
  Google Scholar
• 20 Gorrell M. The 21st Century Searcher: How the Growth of Search Engines Affected the Redesign of EBSCOhost. Against the Grain 2008; 20 (03) 22-26.
  PubMedGoogle Scholar
• 21 Sharma SK, Chen R, Zhang J. Examining Usability of E-learning Systems-An Exploratory Study (Re-search-in-Progress). International Proceedings of Economics Development and Research 2014; 81: 120.
  PubMedGoogle Scholar
• 22 Yan J, El Ahmad A.S. Usability of CAPTCHAs or usability issues in CAPTCHA design. 4th symposium on Usable privacy and security. 2008. Pittsburgh, PA: ACM;
  Google Scholar
• 23 Jonsson A. Usability in three generations business support systems: Assessing perceived usability in the banking industry. Norrköping: Linköping University; 2013
  Google Scholar
• 24 Aljohani M, Blustein J. Heuristic evaluation of university institutional repositories based on DSpace. International Conference of Design, User Experience, and Usability Springer International Publishing; 2015: 119-130.
  Google Scholar
• 25 Tapani J. Game Usability in North American Video Game Industry. Oulu: University of Oulu; 2016
  Google Scholar
• 26 Lilholt PH, Jensen MH, Hejlesen OK. Heuristic evaluation of a telehealth system from the Danish Tele-Care North Trial. Int J Med Inform 2015; 84 (05) 319-326.
  CrossrefPubMedGoogle Scholar
• 27 Lu JM, Hsu YL, Lu CH, Hsu PE, Chen YW, Wang JA. Development of a telepresence robot to rebuild the physical face-to-face interaction in remote interpersonal communication. International Design Alliance (IDA) Congress Education Congress. October, 24–26th; Taipei, Taiwan; 2011
  PubMedGoogle Scholar
• 28 Sparkes J, Valaitis R, McKibbon A. A usability study of patients setting up a cardiac event loop recorder and BlackBerry gateway for remote monitoring at home. Telemedicine and E-health 2012; 18 (06) 484-490.
  CrossrefPubMedGoogle Scholar
• 29 Ozok AA, Wu H, Garrido M, Pronovost PJ, Gurses AP. Usability and perceived usefulness of Personal Health Records for preventive health care: a case study focusing on patients’ and primary care providers’ perspectives. Appl Ergon 2014; 45 (03) 613-628.
  CrossrefPubMedGoogle Scholar
• 30 Zaidi ST, Marriott JL. Barriers and Facilitators to Adoption of a Web-based Antibiotic Decision Support System. South Med Rev 2012; 5 (02) 42-50.
  PubMedGoogle Scholar
• 31 Carayon P, Cartmill R, Blosky MA, Brown R, Hackenberg M, Hoonakker P, Hundt AS, Norfolk E, Wetter-neck TB, Walker JM. ICU nurses’ acceptance of electronic health records. J Am Med Inform Assoc 2011; 18 (06) 812-819.
  CrossrefPubMedGoogle Scholar
• 32 Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010; 43 (01) 159-172.
  CrossrefPubMedGoogle Scholar
• 33 Sun HM, Li SP, Zhu YQ, Hsiao B. The effect of user‘s perceived presence and promotion focus on usability for interacting in virtual environments. Appl Ergon 2015; 50: 126-132.
  CrossrefPubMedGoogle Scholar
• 34 Koppel R. Is healthcare information technology based on evidence?. Yearb Med Inform 2013; 8 (01) 7-12.
  PubMedGoogle Scholar
• 35 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-1203.
  CrossrefPubMedGoogle Scholar
• 36 Stead WW, Lin HS. Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions. Washington (DC): 2009
  Google Scholar
• 37 Koch SH, Weir C, Haar M, Staggers N, Agutter J, Gorges M, Westenskow D. Intensive care unit nurses’ information needs and recommendations for integrated displays to improve nurses’ situation awareness. J Am Med Inform Assoc 2012; 19 (04) 583-590.
  CrossrefPubMedGoogle Scholar
• 38 Lurio J, Morrison FP, Pichardo M, Berg R, Buck MD, Wu W, Kitson K, Mostashari F, Calman N. Using electronic health record alerts to provide public health situational awareness to clinicians. J Am Med Inform Assoc 2010; 17 (02) 217-219.
  CrossrefPubMedGoogle Scholar
• 39 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353.
  CrossrefPubMedGoogle Scholar
• 40 Lopez KD, Febretti A, Stifter J, Johnson A, Wilkie DJ, Keenan G. Toward a More Robust and Efficient Usability Testing Method of Clinical Decision Support for Nurses Derived From Nursing Electronic Health Record Data. Int J Nurs Knowl. 2016 Jun 24.
  PubMedGoogle Scholar
• 41 Zhang J, Walji MF. TURF: toward a unified framework of EHR usability. J Biomed Inform 2011; 44 (06) 1056-1067.
  CrossrefPubMedGoogle Scholar
• 42 Sousa VE, Lopez KD, Febretti A, Stifter J, Yao Y, Johnson A, Wilkie DJ, Keenan GM. Use of Simulation to Study Nurses’ Acceptance and Nonacceptance of Clinical Decision Support Suggestions. Comput Inform Nurs 2015; 33 (10) 465-472.
  CrossrefPubMedGoogle Scholar
• 43 Christensen BL, Kockrow EO. Adult health nursing. St. Louis: Elsevier Health Sciences; 2014
  Google Scholar
• 44 Albertazzi D, Okimoto ML, Ferreira MG. Developing an usability test to evaluate the use of augmented reality to improve the first interaction with a product. Work 2011; 41: 1160-1163.
  PubMedGoogle Scholar
• 45 Freire LL, Arezes PM, Campos JC. A literature review about usability evaluation methods for e-learning platforms. Work-Journal of Prevention Assessment and Rehabilitation 2012; 41: 1038.
  PubMedGoogle Scholar
• 46 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422.
  CrossrefPubMedGoogle Scholar
• 47 Santos NA, Sousa VEC, Pascoal LM, Lopes MVO, Pagliuca LMF. editors. Concept analysis of usability of healthcare technologies. 17th National Nursing Research Seminar. 2013. Natal, RN, Brazil: Brazilian Nursing Association;
  Google Scholar
• 48 Lei J, Xu L, Meng Q, Zhang J, Gong Y. The current status of usability studies of information technologies in China: a systematic study. Biomed Res Int 2014; 2014: 568303.
  PubMedGoogle Scholar
• 49 Lyles CR, Sarkar U, Osborn CY. Getting a technology-based diabetes intervention ready for prime time: a review of usability testing studies. Curr Diab Rep 2014; 14 (10) 534.
  CrossrefPubMedGoogle Scholar
• 50 Wootten AC, Abbott JAM, Chisholm K, Austin DW, Klein B, McCabe M, Murphy DG, Costello AJ. Development, feasibility and usability of an online psychological intervention for men with prostate cancer: My Road Ahead. Internet Interventions 2014; 1 (04) 188-195.
  CrossrefPubMedGoogle Scholar
• 51 Friedman K, Noyes J, Parkin CG. 2-Year follow-up to STeP trial shows sustainability of structured self-monitoring of blood glucose utilization: results from the STeP practice logistics and usability survey (STeP PLUS). Diabetes Technol Ther 2013; 15 (04) 344-347.
  CrossrefPubMedGoogle Scholar
• 52 Stafford E, Hides L, Kavanagh DJ. The acceptability, usability and short-term outcomes of Get Real: A web-based program for psychotic-like experiences (PLEs). Internet Interventions 2015; 2: 266-271.
  CrossrefPubMedGoogle Scholar
• 53 Shah N, Jonassaint J, De Castro L. Patients welcome the Sickle Cell Disease Mobile Application to Record Symptoms via Technology (SMART). Hemoglobin 2014; 38 (02) 99-103.
  CrossrefPubMedGoogle Scholar
• 54 Shyr C, Kushniruk A, Wasserman WW. Usability study of clinical exome analysis software: top lessons learned and recommendations. J Biomed Inform 2014; 51: 129-136.
  CrossrefPubMedGoogle Scholar
• 55 Anders S, Albert R, Miller A, Weinger MB, Doig AK, Behrens M, Agutter J. Evaluation of an integrated graphical display to promote acute change detection in ICU patients. Int J Med Inform 2012; 81 (12) 842-851.
  CrossrefPubMedGoogle Scholar
• 56 Yen PY, Gorman P. Usability testing of digital pen and paper system in nursing documentation. AMIA Annu Symp Proc 2005: 844-848.
  PubMedGoogle Scholar
• 57 McDermott RJ, Sarvela PD. Health education evaluation and measurement: A practitioner‘s perspective. 2 ed. Dubuque, Iowa: William C Brown Pub; 1998
  Google Scholar
• 58 Taveira-Gomes T, Saffarzadeh A, Severo M, Guimaraes MJ, Ferreira MA. A novel collaborative e-learning platform for medical students –ALERT STUDENT. BMC Med Educ 2014; 14: 143.
  CrossrefPubMedGoogle Scholar
• 59 Yui BH, Jim WT, Chen M, Hsu JM, Liu CY, Lee TT. Evaluation of computerized physician order entry system-a satisfaction survey in Taiwan. J Med Syst 2012; 36 (06) 3817-3824.
  CrossrefPubMedGoogle Scholar
• 60 Amrein K, Kachel N, Fries H, Hovorka R, Pieber TR, Plank J, Wenger U, Lienhardt B, Maggiorini M. Glucose control in intensive care: usability, efficacy and safety of Space GlucoseControl in two medical European intensive care units. BMC Endocr Disord 2014; 14: 62.
  CrossrefPubMedGoogle Scholar
• 61 Ratwani RM, Hettinger AZ, Fairbanks RJ. Barriers to comparing the usability of electronic health records. J Am Med Inform Assoc. 2016 DOI:10.1093/jamia/ocw117
  PubMedGoogle Scholar
• 62 21st Century Cures Act, H.R.6. To accelerate the discovery, development, and delivery of 21st century cures, and for other purposes; 2015.
  PubMed
• 63 Hesselink G, Kuis E, Pijnenburg M, Wollersheim H. Measuring a caring culture in hospitals: a systematic review of instruments. BMJ Open 2013; 3 (09) e003416.
  CrossrefPubMedGoogle Scholar
• 64 Cox CE, Wysham NG, Walton B, Jones D, Cass B, Tobin M, Jonsson M, Kahn JM, White DB, Hough CL, Lewis CL, Carson SS. Development and usability testing of a Web-based decision aid for families of patients receiving prolonged mechanical ventilation. Ann Intensive Care 2015; 5: 6.
  CrossrefPubMedGoogle Scholar
• 65 Im EO, Chee W. editors. Evaluation of the decision support computer program for cancer pain management. Oncology nursing forum 2006. Pittsburgh, PA: Oncology Nursing Society;
  Google Scholar
• 66 Kim MS, Shapiro JS, Genes N, Aguilar MV, Mohrer D, Baumlin K, Belden JL. A pilot study on usability analysis of emergency department information system by nurses. Appl Clin Inform 2012; 3 (01) 135-153.
  Article in Thieme ConnectPubMedGoogle Scholar
• 67 Kobak KA, Stone WL, Wallace E, Warren Z, Swanson A, Robson K. A web-based tutorial for parents of young children with autism: results from a pilot study. Telemed J E Health 2011; 17 (10) 804-808.
  CrossrefPubMedGoogle Scholar
• 68 Levin ME, Pistorello J, Seeley JR, Hayes SC. Feasibility of a prototype web-based acceptance and commitment therapy prevention program for college students. J Am Coll Health 2014; 62 (01) 20-30.
  CrossrefPubMedGoogle Scholar
• 69 Lin CA, Neafsey PJ, Strickler Z. Usability testing by older adults of a computer-mediated health communication program. J Health Commun 2009; 14 (02) 102-118.
  CrossrefPubMedGoogle Scholar
• 70 Saleem JJ, Haggstrom DA, Militello LG, Flanagan M, Kiess CL, Arbuckle N, Doebbeling BN. Redesign of a computerized clinical reminder for colorectal cancer screening: a human-computer interaction evaluation. BMC Med Inform Decis Mak 2011; 11: 74.
  CrossrefPubMedGoogle Scholar
• 71 Schnall R, Cimino JJ, Bakken S. Development of a prototype continuity of care record with context-specific links to meet the information needs of case managers for persons living with HIV. Int J Med Inform 2012; 81 (08) 549-555.
  CrossrefPubMedGoogle Scholar
• 72 Schutte J, Gales S, Filippone A, Saptono A, Parmanto B, McCue M. Evaluation of a telerehabilitation system for community-based rehabilitation. Int J Telerehabil 2012; 4 (01) 15-24.
  PubMedGoogle Scholar
• 73 Sharma P, Dunn RL, Wei JT, Montie JE, Gilbert SM. Evaluation of point-of-care PRO assessment in clinic settings: integration, parallel-forms reliability, and patient acceptability of electronic QOL measures during clinic visits. Qual Life Res 2016; 25 (03) 575-583.
  CrossrefPubMedGoogle Scholar
• 74 Shoup JA, Wagner NM, Kraus CR, Narwaney KJ, Goddard KS, Glanz JM. Development of an Interactive Social Media Tool for Parents With Concerns About Vaccines. Health Educ Behav 2015; 42 (03) 302-312.
  CrossrefPubMedGoogle Scholar
• 75 Staggers N, Kobus D, Brown C. Nurses’ evaluations of a novel design for an electronic medication administration record. Comput Inform Nurs 2007; 25 (02) 67-75.
  CrossrefPubMedGoogle Scholar
• 76 Stellefson M, Chaney B, Chaney D, Paige S, Payne-Purvis C, Tennant B, Walsh-Childers K, Sriram P, Alber J. Engaging community stakeholders to evaluate the design, usability, and acceptability of a chronic obstructive pulmonary disease social media resource center. JMIR Res Protoc 2015; 4 (01) e17.
  CrossrefPubMedGoogle Scholar
• 77 Bozkurt S, Zayim N, Gulkesen KH, Samur MK, Karaagaoglu N, Saka O. Usability of a web-based personal nutrition management tool. Inform Health Soc Care 2011; 36 (04) 190-205.
  CrossrefPubMedGoogle Scholar
• 78 Fritz F, Balhorn S, Riek M, Breil B, Dugas M. Qualitative and quantitative evaluation of EHR-integrated mobile patient questionnaires regarding usability and cost-efficiency. Int J Med Inform 2012; 81 (05) 303-313.
  CrossrefPubMedGoogle Scholar
• 79 Meldrum D, Glennon A, Herdman S, Murray D, McConn-Walsh R. Virtual reality rehabilitation of balance: assessment of the usability of the Nintendo Wii((R)) Fit Plus. Disabil Rehabil Assist Technol 2012; 7 (03) 205-210.
  CrossrefPubMedGoogle Scholar
• 80 Memedi M, Westin J, Nyholm D, Dougherty M, Groth T. A web application for follow-up of results from a mobile device test battery for Parkinson‘s disease patients. Comput Meth Prog Bio 2011; 104 (02) 219-226.
  CrossrefPubMedGoogle Scholar
• 81 Ozel D, Bilge U, Zayim N, Cengiz M. A web-based intensive care clinical decision support system: from design to evaluation. Inform Health Soc Care 2013; 38 (02) 79-92.
  CrossrefPubMedGoogle Scholar
• 82 Oztekin A, Kong ZJ, Uysal O. UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis. International Journal of Industrial Ergonomics 2010; 40: 455-469.
  CrossrefPubMedGoogle Scholar
• 83 Stjernsward S, Ostman M. Illuminating user experience of a website for the relatives of persons with depression. Int J Soc Psychiatry 2011; 57 (04) 375-386.
  CrossrefPubMedGoogle Scholar
• 84 Van der Weegen S, Verwey R, Tange HJ, Spreeuwenberg MD, de Witte LP. Usability testing of a monitoring and feedback tool to stimulate physical activity. Patient Prefer Adherence 2014; 8: 311-322.
  PubMedGoogle Scholar
• 85 Hao AT, Wu LP, Kumar A, Jian WS, Huang LF, Kao CC, Hsu CY. Nursing process decision support system for urology ward. Int J Med Inform 2013; 82 (07) 604-612.
  CrossrefPubMedGoogle Scholar
• 86 Huang H, Lee TT. Evaluation of ICU nurses’ use of the clinical information system in Taiwan. Comput Inform Nurs 2011; 29 (04) 221-229.
  CrossrefPubMedGoogle Scholar
• 87 Lacerda TC, von Wangenheim CG, von Wangenheim A, Giuliano I. Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context. Journal of Biomedical Informatics 2014; 52: 222-230.
  CrossrefPubMedGoogle Scholar
• 88 Lee TT, Mills ME, Bausell B, Lu MH. Two-stage evaluation of the impact of a nursing information system in Taiwan. Int J Med Inform 2008; 77 (10) 698-707.
  CrossrefPubMedGoogle Scholar
• 89 Moattari M, Moosavinasab E, Dabbaghmanesh MH, ZarifSanaiey N. Validating a Web-based Diabetes Education Program in continuing nursing education: knowledge and competency change and user perceptions on usability and quality. J Diabetes Metab Disord 2014; 13: 70.
  CrossrefPubMedGoogle Scholar
• 90 Peikari HR, Shah MH, Zakaria MS, Yasin NM, Elhissi A. The impacts of second generation e-prescribing usability on community pharmacists outcomes. Res Social Adm Pharm 2015; 11 (03) 339-351.
  CrossrefPubMedGoogle Scholar
• 91 Sawka AM, Straus S, Gafni A, Meiyappan S, O’Brien MA, Brierley JD, Tsang RW, Rotstein L, Thabane L, Rodin G, George SR, Goldstein DP. A usability study of a computerized decision aid to help patients with, early stage papillary thyroid carcinoma in, decision-making on adjuvant radioactive iodine treatment. Patient Educ Couns 2011; 84 (02) e24-e27.
  CrossrefPubMedGoogle Scholar
• 92 Heikkinen K, Suomi R, Jaaskelainen M, Kaljonen A, Leino-Kilpi H, Salantera S. The creation and evaluation of an ambulatory orthopedic surgical patient education web site to support empowerment. Comput Inform Nurs 2010; 28 (05) 282-290.
  CrossrefPubMedGoogle Scholar
• 93 Jeon E, Park HA. Development of a smartphone application for clinical-guideline-based obesity management. Healthc Inform Res 2015; 21 (01) 10-20.
  CrossrefPubMedGoogle Scholar
• 94 Li LC, Adam PM, Townsend AF, Lacaille D, Yousefi C, Stacey D, Gromala D, Shaw CD, Tugwell P, Back-man CL. Usability testing of ANSWER: a web-based methotrexate decision aid for patients with rheumatoid arthritis. BMC Med Inform Decis Mak 2013; 13: 131.
  CrossrefPubMedGoogle Scholar
• 95 Joshi A, Amadi C, Trout K, Obaro S. Evaluation of an interactive surveillance system for monitoring acute bacterial infections in Nigeria. Perspect Health Inf Manag 2014; 11: 1f.
  PubMedGoogle Scholar
• 96 Albu M, Atack L, Srivastava I. Simulation and gaming to promote health education: Results of a usability test. Health Education Journal. 2015 74(2).
  PubMedGoogle Scholar
• 97 Daniels JP, King AD, Cochrane DD, Carr R, Shaw NT, Lim J, Ansermino JM. A human factors and survey methodology-based design of a web-based adverse event reporting system for families. Int J Med Inform 2010; 79 (05) 339-348.
  CrossrefPubMedGoogle Scholar
• 98 Lewis JR. IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. Int J Hum-Comput Int 1995; 7 (01) 57-78.
  PubMedGoogle Scholar
• 99 Lewis JR. Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ. Human Factors Society 36th Annual Meeting; Atlanta: Human Factors and Ergonomics Society; 1992; 1259-1263.
  PubMedGoogle Scholar
• 100 Chin JP, Diehl VA, Norman KL. editors. Development of an instrument measuring user satisfaction of the human-computer interface. SIGCHI conference on Human factors in computing systems. 1988
  PubMedGoogle Scholar
• 101 Wilkinson A, Forbes A, Bloomfield J, Fincham Gee C. An exploration of four web-based open and flexible learning modules in post-registration nurse education. Int J Nurs Stud 2004; 41 (04) 411-424.
  CrossrefPubMedGoogle Scholar
• 102 Lewis JR. Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ. ACM Sigchi Bulletin 1991; 23 (Suppl. 01) 78-81.
  PubMedGoogle Scholar
• 103 Brooke J. SUS: A quick and dirty usability scale. Usability evaluation in industry 1996; 189 (Suppl. 194) 4-7.
  PubMedGoogle Scholar
• 104 Lewis JR, Henry SC, Mack RL. Integrated office software benchmarks: A case study. In: Dapler D. editor. Interact 90 –3rd IFIP International Conference on Human-Computer Interaction. Cambridge; UK: North-Holland: 1990: 337-243.
  Google Scholar
• 105 Lewis JR. Psychometric evaluation of the computer system usability questionnaire: The CSUQ (Tech. Report 54.723). Boca Raton, FL: International Business Machines Corporation; 1992
  Google Scholar
• 106 Frytak J. Measurement. In: Kane R. editor. Understanding health care outcomes research. Gaithersburg, MD: Aspen Publications; 1997: 213-237.
  Google Scholar
• 107 Lewis JR, Sauro J. The Factor Structure of the System Usability Scale. In: Kurosu M. editor. Human Centered Design. Heidelberg: Springer; 2009: 94-103.
  Google Scholar
• 108 Streiner DL. Starting at the beginning: An introduction to coefficient alpha and internal consistency. J Pers Assess 2003; 80 (Suppl. 01) 99-103.
  CrossrefPubMedGoogle Scholar
• 109 Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. Journal of Human–Computer Interaction 2008; 24 (Suppl. 06) 574-594.
  PubMedGoogle Scholar
• 110 Bland JM, Altman DG. Statistics notes: Cronbach‘s alpha. BMJ 1997; 314: 572.
  CrossrefPubMedGoogle Scholar
• 111 Streiner DL, Norman GR. Health measurement scales 2ed. New York: Oxford University Press; 1995
  Google Scholar
• 112 Kass RA, Tinsley HEA. Factor analysis. J Leisure Res 1979; 11: 120-138.
  PubMedGoogle Scholar
• 113 Brooke J. SUS: a retrospective. Journal of usability studies 2013; 8 (Suppl. 02) 29-40.
  PubMedGoogle Scholar
• 114 Yen PY, Wantland D, Bakken S. Development of a Customizable Health IT Usability Evaluation Scale. AMIA Annu Symp Proc 2010; 2010: 917-921.
  PubMedGoogle Scholar
• 115 Yen PY, Sousa KH, Bakken S. Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results. J Am Med Inform Assoc 2014; 21 (e2) e241-8.
  CrossrefPubMedGoogle Scholar
• 116 Sousa VE, Matson J, Lopez KD. Questionnaire Adapting Little Changes Mean a Lot. Western J Nurs Res. 2016 0193945916678212.
  PubMedGoogle Scholar
• 117 Lin MI, Hong RH, Chang JH, Ke XM. Usage Position and Virtual Keyboard Design Affect Upper-Body Kinematics, Discomfort, and Usability during Prolonged Tablet Typing. PLoS One. 2015 10(12): e0143585.
  PubMedGoogle Scholar
• 118 Koppel R, Leonard CE, Localio AR, Cohen A, Auten R, Strom BL. Identifying and quantifying medication errors: evaluation of rapidly discontinued medication orders submitted to a computerized physician order entry system. J Am Med Inform Assoc 2008; 15 (Suppl. 04) 461-465.
  CrossrefPubMedGoogle Scholar
• 119 Ahmed A, Chandra S, Herasevich V, Gajic O, Pickering BW. The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. Crit Care Med 2011; 39 (Suppl. 07) 1626-1634.
  CrossrefPubMedGoogle Scholar
• 120 Duong M, Bertin K, Henry R, Singh D, Timmins N, Brooks D, Mathur S, Ellerton C. Developing a physiotherapy-specific preliminary clinical decision-making tool for oxygen titration: a modified delphi study. Physiother Can 2014; 66 (Suppl. 03) 286-295.
  CrossrefPubMedGoogle Scholar
• 121 Vuong AM, Huber Jr. JC, Bolin JN, Ory MG, Moudouni DM, Helduser J, Begaye D, Bonner TJ, Forjuoh SN. Factors affecting acceptability and usability of technological approaches to diabetes self-management: a case study. Diabetes Technol Ther 2012; 14 (Suppl. 12) 1178-1182.
  CrossrefPubMedGoogle Scholar
• 122 Jokela T, Iivari N, Matero J, Karukka M. editors. The standard of user-centered design and the standard definition of usability: analyzing ISO 13407 against ISO 9241–11. The Latin American conference on Human-computer interaction; 2003. New York: ACM;
  Google Scholar
• 123 Anastasi A. Psychological testing. New York: Macmillan; 1988
  Google Scholar
• 124 Parke CS. Missing data. In: Parke CS. editor. Essential first steps to data analysis: Scenario-based examples using SPSS. Thousand Oaks: Sage Publications; 2012: 179.
  Google Scholar
• 125 Gaber J. Face validity. In: Salkind NJ. editor. Encyclopedia of Research Design. Thousand Oaks: Sage Publications; 2010: 472-475.
  Google Scholar
• 126 Johnson EK, Nelson CP. Utility and Pitfalls in the Use of Administrative Databases for Outcomes Assessment. The Journal of Urology 2013; 190 (Suppl. 01) 17-18.
  CrossrefPubMedGoogle Scholar
• 127 Fratello J, Kapur TD, Chasan A. Measuring success: A guide to becoming an evidence-based practice. New York: Vera Institute of Justice; 2013
  Google Scholar
• 128 Greenhalgh J, Long AF, Brettle AJ, Grant MJ. Reviewing and selecting outcome measures for use in routine practice. J Eval Clin Pract 1998; 4 (04) 339-350.
  CrossrefPubMedGoogle Scholar
• 129 Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, Bakken S, Kaplan CP, Squiers L, Fabrizio C, Fernandez M. How we design feasibility studies. American Journal of Preventive Medicine 2009; 36 (Suppl. 05) 452-457.
  CrossrefPubMedGoogle Scholar
• 130 Madrigal D, McClain B. Strengths and weaknesses of quantitative and qualitative research. UXmatters; 2012 Available from: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php
  PubMedGoogle Scholar

• Supplementary Material