Abstract
The inadequacy of infrastructure for bringing mobile healthcare apps from developers to clinical practitioners has kept the 165,000+ currently available healthcare apps from integration into routine clinical practice. The absence of regulatory and certification processes and the unlikelihood that many apps will be tested with credible clinical trials leaves it up to expert reviews to lead clinicians to high-quality apps. However, most app reviews are not collected in an easily searchable location that facilitates comparison of the merits of alternative apps, and surveys of existing expert reviews reveal a lack of standards for objective and reliable assessments. Furthermore, most published recommendations for apps are not based on their validity or appropriateness for clinical use.This article describes development of the Interactive Mobile App Review Toolkit (IMART), a technology-assisted system for producing verifiable app reviews intended for clinicians and its accompanying evidence-based thesaurus of standards. IMART will present systematized reviews in a searchable, curated library where clinicians can find and compare reviews about apps that are tagged as to the treatment needs of clients/patients and that address how the apps could be integrated into the visitor’s clinical practice.The assessment criteria in the “Digital Health Standards Thesaurus” can be used in reviewing apps and other digital health products, in developing apps, in making decisions about investing venture capital or foundation funds into concepts for new digital health products, and in selecting such products into the “formularies” of third party payers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Change history
01 December 2017
On page 5 the first line in the body of Table 1 should read:
References
Abraham, C., & Michie, S. (2008). A taxonomy of behavior change techniques used in interventions. Health Psychology, 27(3), 379–387. doi:https://doi.org/10.1037/0278-6133.27.3.379.
Agarwal, S., LeFevre, A. E., Lee, J., L’Engle, K., Mehl, G., Sinha, C., & Labrique, A. (2016). Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ, 352, i1174. doi:https://doi.org/10.1136/bmj.i1174.
Aitken, M., & Lyle, J. (2015). Patient adoption of mHealth: use, evidence and remaining barriers to mainstream acceptance. Parsippany: IMS Institute for Healthcare Informatics Retrieved fromhttps://doi.org/www.imshealth.com/files/web/IMSH%20Institute/Reports/Patient%20Adoption%20of%20mHealth/IIHI_Patient_Adoption_of_mHealth.pdf.
Aladwani, A. M., & Palvia, P. C. (2002). Developing and validating an instrument for measuring user-perceived web quality. Information & Management, 39(6), 467–476. doi:https://doi.org/10.1016/S0378-7206(01)00113-6.
American Medical Association (Producer). (2016, June 11). 2016 AMA Annual Meeting—AMA EVP/CEO James L. Madara, MD. [Video]. Retrieved from https://doi.org/www.youtube.com/watch?v=RbWgJIOWQmQ.
American Society for Quality. ASQ health care quality & patient experience survey: summary report. Retrieved from https://doi.org/asq.org/public/healthcare/asq-healthcare-quality-patient-summary-report.pdf.
Armontrout, J., Torous, J., Fisher, M., Drogin, E., & Gutheil, T. (2016). Mobile mental health: navigating new rules and regulations for digital tools. Current Psychiatry Reports. doi:https://doi.org/10.1007/s11920-016-0726-x.
Australian Government Department of Health. (2013). Therapeutic goods administration. Regulation of medical software and mobile medical ‘apps’. Retrieved from https://doi.org/www.tga.gov.au/regulation-medical-software-and-mobile-medical-apps.
Bakker, D., Kazantzis, N., Rickwood, D., & Rickard, N. (2016). Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Mental Health, 1(3), e7. doi:https://doi.org/10.2196/mental.4984.
BinDhim, N. F., Hawkey, A., & Trevena, L. (2015). A systematic review of quality assessment methods for smartphone health apps. Telemedicine and eHealth, 21(2), 97104. doi:https://doi.org/10.1089/tmj.2014.0088.
Blenner, S. R., Köllmer, M., Rouse, A. J., Daneshvar, N., Williams, C., & Andrews, L. B. (2016). Privacy policies of android diabetes apps and sharing of health information. JAMA. the Journal of the American Medical Association, 315(10), 1051–1052. doi:https://doi.org/10.1001/jama.2015.19426.
Boulware, L. E., Cooper, L. A., Ratner, L. E., LaVeist, T. A., & Powe, N. R. (2003). Race and trust in the health care system. Public Health Reports, 118(4), 358–365.
Cairns, P. (2013). A commentary on short questionnaires for assessing usability. Interacting with Computers, 25(4), 312–316. doi:https://doi.org/10.1093/iwc/iwt019.
Centers for Disease Control and Prevention. (2015). Characteristics of an effective health education curriculum. Retrieved from https://doi.org/www.cdc.gov/healthyschools/sher/characteristics/index.htm.
Chan, S. R., & Misra, S. (2014). Certification of mobile apps for health care. JAMA. The Journal of the American Medical Association, 312(11), 1155–1156. doi:https://doi.org/10.1001/jama.2014.90.
Chan, S., Torous, J., Hinton, L., & Yellowlees, P. (2015). Towards a framework for evaluating mobile mental health apps. Telemedicine and e-Health, 21(12), 1038–1041. doi:https://doi.org/10.1089/tmj.2015.000.
Chen, J., Cade, J. E., & Allman-Farinelli, M. (2015). The most popular smartphone apps for weight loss: a quality assessment. JMIR mHealth and uHealth, 3(4), e104. doi:https://doi.org/10.2196/mhealth.4334.
Cheung, C. M., Sia, C., & Kuan, K. K. Y. (2012). Is this review believable? A study of factors affecting the credibility of online consumer reviews from an ELM perspective. Journal of the Association for Information Systems, 13(8), 618–635 Retrieved from https://doi.org/aisel.aisnet.org/jais/vol13/iss8/2.
Commons, M. L. (2007). Introduction to the model of hierarchical complexity. Behavioral Development Bulletin, 13(1), 1–6. doi:https://doi.org/10.1037/h0100493.
Commons, M. L., & Pekker, A. (2007). Hierarchical complexity: a formal theory. Manuscript submitted for publication. Retrieved from https://doi.org/www.tiac.net/~commons/Hierarchical%20Complexity%20-%20A%20Formal%20Theory%20(Commons%20&%20Pekker).rtf.
Dehling, T., Gao, F., Schneider, S., & Sunyaev, A. (2015). Exploring the far side of mobile health: information security and privacy of mobile health apps on iOS and Android. JMIR mHealth and uHealth, 19(3), e8. doi:https://doi.org/10.2196/mhealth.3672.
Devine, T., Broderick, J., Harris, L. M., Wu, H., & Hilfiker, S. W. (2016). Making quality health websites a national public health priority: toward quality standards. JMIR, 18(8), e211. doi:https://doi.org/10.2196/jmir.5999.
Farag, S., Fields, J., Pereira, E., Chyjek, K., & Chen, K. T. (2016). Identification and rating of gynecologic oncology applications using the APPLICATIONS scoring system. Telemedicine and e-Health, 127(137S), 1–7. doi:https://doi.org/10.1097/01.aog.0000483553.19111.9e.
Finstad, K. (2010). The usability metric for user experience. Interacting with Computers, 22(5), 323–327. doi:https://doi.org/10.1016/j.intcom.2010.04.004.
FPF Mobile Apps Study. (2016). Washington, DC: future of privacy forum. Retrieved from https://doi.org/wp-content/uploads/2016/08/2016-FPF-Mobile-Apps-Study_final.pdf.
Girardello, A., & Michahelles, F. (2010). APPAware: which mobile applications are hot. Association for Computing Machinery 12th International Conference on Human Computer Interaction with Mobile Devices and Services, Lisbon, Portugal. 431–434. doi:https://doi.org/10.1145/1851600.1851698.
Glenn, T., & Monteith, S. (2014). Privacy in the digital world: medical and health data outside of HIPAA protections. Current Psychiatry Reports, 16(11), 494. doi:https://doi.org/10.1007/s11920-014-0494-4.
Graafland, M., Dankbaar, M., Mert, A., Lagro, J., De Wit-Zuurendonk, L., Schuit, S., et al. (2014). How to systematically assess serious games applied to health care. JMIR Serious Games, 2(2), e11. doi:https://doi.org/10.2196/games.3825.
Hall, J. L., & McGraw, D. (2014). For telehealth to succeed, privacy and security risks must be identified and addressed. Health Affairs, 33(2), 216–221. doi:https://doi.org/10.1377/hlthaff.2013.0997.
Hillix, W. A., & L’Abate, L. (2012). Chapter 1. The role of paradigms in science and theory construction. In L. L’Abate (Ed.), Paradigms in theory construction (pp. 3–18). New York: Springer. doi:https://doi.org/10.1007/978-1-4614-0914-4.
Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P., & Car, J. (2015). Unaddressed privacy risks in accredited health and wellness apps: a cross-sectional systematic assessment. BMC Medicine, 13, 214. doi:https://doi.org/10.1186/s12916-015-0444-y.
Huguet, A., Rao, S., McGrath, P. J., Wozney, L., Wheaton, M., Conrod, J., et al. (2016). A systematic review of cognitive behavioral therapy and behavioral activation apps for depression. PloS One, 11(5), e0154248. doi:https://doi.org/10.1371/journal.pone.0154248.
Kay-Lambkin, F., White, A., Baker, A. L., Kavanagh, D. J., Klein, B., Proudfoot, J., et al. (2011). Assessment of function and clinical utility of alcohol and other drug web sites: an observational, qualitative study. BMC Public Health, 11(1), 277. doi:https://doi.org/10.1186/1471-2458-11-277.
Klosowski, T. (2014). Lots of health apps are selling your data. Here’s why. Retrieved from https://doi.org/lifehacker.com/lots-of-health-apps-are-selling-your-data-heres-why-1574001899.
Kuehnhausen, M., & Frost, V. S. (2013). Trusting smartphone apps? To install or not to install, that is the question (pp. 30–37). San Diego: IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA). doi:https://doi.org/10.1109/CogSIMA.2013.6523820.
Kuhn, T. S. (2012). The structure of scientific revolutions (4th ed.). Chicago, IL: University of Chicago Press.
L’Abate, L. (2015a). Review of “science and pseudoscience in clinical psychology (2nd ed.)”, edited by Scott O. Lilienfeld, Steven J. Lynn, and Jeffrey M. Lohr. The American Journal of Family Therapy, 43(2), 210–211. doi:https://doi.org/10.1080/01926187.2014.1002365.
L’Abate, L. (2015b). Chapter 6. Beyond systems thinking: toward a unifying framework for human relationships. In L. L’Abate (Ed.), Concreteness and specificity in clinical psychology. Evaluations and interventions (pp. 73–89). New.York: Springer International Publishing. doi:https://doi.org/10.1007/978-3-319-13284-6_6.
Lamichhane, D., & Armstrong, M. (2015). Systematic review of Parkinson’s disease related mobile applications. Neurology, 84(14), Supplement P7.293.
Lavie, T., & Tractinsky, N. (2004). Assessing dimensions of perceived visual aesthetics of web sites. International Journal of Human-Computer Studies, 60(3), 269–298. doi:https://doi.org/10.1016/j.ijhcs.2003.09.002.
Lewis, T. L., & Wyatt, J. C. (2014). mHealth and mobile medical apps: a framework to assess risk and promote safer use. Journal of Medical Internet Research, 16(9), e21. doi:https://doi.org/10.2196/jmir.3133.
Lorence, D. P., & Greenberg, L. (2006). The zeitgeist of online health search. Journal of General Internal Medicine, 21(2), 134–139. doi:https://doi.org/10.1007/s11606-006-0247-0.
Maheu, M. M., Pulier, M. L., & Roy, S. (2013). Finding, evaluating, and using smartphone applications. In G. P. Koocher, J. C. Norcross, & B. A. Breene (Eds.), Psychologists’ desk reference (Third ed., pp. 704–708). New York: Oxford University Press.
Mendelsohn, D. (2012). A critic’s manifesto. Retrieved from https://doi.org/www.newyorker.com/books/page-turner/a-critics-manifesto.
Mendiola, M. F., Kalnicki, M., & Lindenauer, S. (2015). Valuable features in mobile health apps for patients and consumers: content analysis of apps and user ratings. JMIR mHealth and uHealth, 3(2), e40. doi:https://doi.org/10.2196/mhealth.4283.
Meurk, C., Leung, J., Hall, W., Head, B. W., & Whiteford, H. (2016). Establishing and governing e-mental health care in Australia: a systematic review of challenges and a call for policy-focused research. Journal of Medical Internet Research, 18(1), e10. doi:https://doi.org/10.2196/jmir.4827.
mHIMSS App Usability Work Group. (2012). Selecting a mobile app: evaluating the usability of medical applications. Retrieved from https://doi.org/s3.amazonaws.com/rdcms-himss/files/production/public/HIMSSguidetoappusabilityv1mHIMSS.pdf.
Moshagen, M., & Thielsch, M. T. (2010). Facets of visual aesthetics. International Journal of Human-Computer Studies, 68(10), 689–709. doi:https://doi.org/10.1016/j.ijhcs.2010.05.006.
Moshagen, M., & Thielsch, M. (2013). A short version of the visual aesthetics of websites inventory. Behaviour & Information Technology, 32(12), 1305–1311. doi:https://doi.org/10.1080/0144929X.2012.694910.
Moustakis, V. S., Litos, C., Dalivigas, A., & Tsironis, L. (2004). Website quality assessment criteria. Cambridge: Proceedings of the Ninth International Conference on Information Quality Retrieved from https://doi.org/mitiq.mit.edu/ICIQ/Documents/IQ%20Conference%202004/Papers/WebsiteQualityAssessmentCriteria.pdf.
Nicholas, J., Larsen, M. E., Proudfoot, J. G., & Christensen, H. (2015). Mobile apps for bipolar disorder: a systematic review of features and content quality. Journal of Medical Internet Research, 17(8), e198. doi:https://doi.org/10.2196/jmir.4581.
Northeastern University. (2016). Android apps can secretly track users’ whereabouts, researchers find. Retrieved from https://doi.org/www.sciencedaily.com/releases/2016/08/160810141939.htm.
Pereira-Azevedo, N., Osório, L., Cavadas, V., Fraga, A., Carrasquinho, E., Cardoso de Oliveira, E., et al. (2016). Expert involvement predicts mHealth app downloads: multivariate regression analysis of urology apps. JMIR mHealth and uHealth, 4(3), e86. doi:https://doi.org/10.2196/mhealth.5738.
Powell, A. C., Landman, A. B., & Bates, D. W. (2014a). In search of a few good apps. JAMA. The Journal of the American Medical Association, 311(18), 1851–1852. doi:https://doi.org/10.1001/jama.2014.2564.
Powell, A. C., Landman, A. B., & Bates, D. W. (2014b). Certification of mobile apps for healthcare—reply. JAMA. The Journal of the American Medical Association, 312(11), 1156–1157. doi:https://doi.org/10.1001/jama.2014.9007.
Powell, A. C., Torous, J., Chan, S., Raynor, G. S., Shwarts, E., Shanahan, M., & Landman, A. B. (2016). Interrater reliability of mhealth app rating measures: analysis of top depression and smoking cessation apps. JMIR mHealth and uHealth, 4(1), e15. doi:https://doi.org/10.2196/mhealth.5176.
Psyberguide. Retrieved from https://doi.org/www.psyberguide.org/.
Reynolds, J., Griffiths, K. M., Cunningham, J. A., Bennett, K., & Bennett, A. (2015). Clinical practice models for the use of e-mental health resources in primary health care by health professionals and peer workers: a conceptual framework. JMIR Mental Health, 2(1), e6. doi:https://doi.org/10.2196/mental.4200.
Robeznieks, A. (2016). Light regulatory touch called appropriate for mobile health apps. Retrieved from https://doi.org/medcitynews.com/2016/07/regulatory-touch-health-apps/.
Rogers, E. (2003). Diffusion of innovations (5th ed.). New York, NY: Simon & Schuster.
Rowe, J. (2016). Policymakers wrestle with intricacies of regulating mobile health tech. Retrieved from https://doi.org/www.himssfuturecare.com/blog/policymakers-wrestle-intricacies-regulating-mobile-health-tech.
Sarkar, U., Gourley, G. I., Lyles, C. R., Tieu, L., Clarity, C., Newmark, L., et al. (2016). Usability of commercially available mobile applications for diverse patients. Journal of General Internal Medicine, online ahead of print. doi:https://doi.org/10.1007/s11606-016-3771-6.
Scher, D. L. (2013), May 7. Should you recommend health apps? Medscape Psychiatry & Mental Health. Retrieved from https://doi.org/www.medscape.com/viewarticle/803503_2.
Schnall, R., Mosley, J. P., Iribarren, S. J., Bakken, S., Carballo-Diéguez, A., & Brown III, W. (2015). Comparison of a user-centered design, self-management app to existing mhealth apps for persons living with HIV. JMIR mHealth and uHealth, 3(3), e91. doi:https://doi.org/10.2196/mhealth.4882.
Shen, N., Levitan, M., Johnson, A., Bender, J. L., Hamilton-Page, M., Jadad, A. R., & Wiljer, D. (2015). Finding a depression app: a review and content analysis of the depression app marketplace. JMIR mHealth and uHealth, 3(1), e16. doi:https://doi.org/10.2196/mhealth.3713.
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015). Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth, 3(1), e27. doi:https://doi.org/10.2196/mhealth.3422.
Stoyanov, S. R., Hides, L., Kavanagh, D. J., & Wilson, H. (2016). Development and validation of the user version of the mobile application rating scale (uMARS). JMIR mHealth uHealth, 4(2), e72. doi:https://doi.org/10.2196/mhealth.5849.
Tomlinson, M., Rotheram-Borus, M. J., Swartz, L., & Tsai, A. C. (2013). Scaling up mHealth: where is the evidence? PLoS Medicine, 10(2), e1001382. doi:https://doi.org/10.1371/journal.pmed.1001382.
Torous, J. B., Chan, S. R., Yellowlees, P., & Borland, R. (2016). To use or not? Evaluating ASPECTS of smartphone apps and mobile technology for clinical care in psychiatry. Journal of Clinical Psychiatry, 77(6), e734–e738. doi:https://doi.org/10.4088/JCP.15com10619.
U.S. Department of Health and Human Services, Food and Drug Administration. (2015). A mobile medical applications guidance for industry and food and drug administration staff. (Document 1741). Retrieved from https://doi.org/www.fda.gov/downloads/MedicalDevices/.../UCM263366.pdf.
U.S. Department of Health and Human Services, Food and Drug Administration. (2016). Examples of MMAs that are NOT medical devices. Retrieved from https://doi.org/www.fda.gov/MedicalDevices/DigitalHealth/MobileMedicalApplications/ucm388746.htm.
Väätäjä, H., Koponen, T., & Roto, V. (2009). Developing practical tools for user experience evaluation: a case from mobile news journalism. European Conference on Cognitive Ergonomics: Designing Beyond the Product—Understanding Activity and User Experience in Ubiquitous Environments, Finland, Article 33. Retrieved from https://doi.org/www.vtt.fi/inf/pdf/symposiums/2009/S258.pdf.
van Vesen, L., Beaujean, D. J. M. A., & van Gemert-Pijnen, J. E. W. C. (2013). Why mobile healthapp overload drives us crazy, and how to restore the sanity. BMC Medical Informatics & Decision Making, 13(23), 1–5. doi:https://doi.org/10.1186/1472-6947-1.
Vermeeren, A. P., Law, E. L., Roto, V., Obrist, M., Hoonhout, J., & Väänänen-Vainio-Mattila, K. (2010). User experience evaluation methods: current state and development needs. NordiCHI ‘10 Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 521–530. doi:https://doi.org/10.1145/1868914.1868973.
von Reischach, F., Dubach, E., Michahelles, F., & Schmidt, A. (2010). An evaluation of product review modalities for mobile phones (pp. 199–208). Lisbon: Association for Computing Machinery 12th International Conference on Human Computer Interaction with Mobile Devices and Services. doi:https://doi.org/10.1145/1851600.1851635.
Wang, A., An, N., Lu, X., Chen, H., Li, C., & Levkoff, S. (2014). A classification scheme for analyzing mobile apps used to prevent and manage disease in late life. JMIR mHealth and uHealth, 2(1), e6. doi:https://doi.org/10.2196/mhealth.2877.
West, J. H., Hall, P. C., Hanson, C. L., Barnes, M. D., Giraud-Carrier, C., & Barrett, J. (2012). There’s an app for that: content analysis of paid health and fitness apps. Journal of Medical Internet Research, 14(3), Je72. doi:https://doi.org/10.2196/jmir.1977.
Wicks, P., & Chiauzzi, E. (2015). “Trust but verify”—five approaches to ensure safe medical apps. BMC Medicine, 13(1), 205. doi:https://doi.org/10.1186/s12916-015-0451-z.
Wilhide III, C., Peeples, M. M., & Anthony Kouyaté, R. C. (2016). Evidence-based mHealth chronic disease mobile app intervention design: development of a framework. JMIR Research Protocols, 5(1), e25. doi:https://doi.org/10.2196/resprot.4838.
Will, L. (2012). The ISO 25964 data model for the structure of an information retrieval thesaurus. Bulletin of the American Society for Information Science and Technology, 38(4), 48–51. doi:https://doi.org/10.1002/bult.2012.1720380413.
AMA Wire. (2016). Deeper dive into digital snake oil: Q&A with Dr. Madara. Retrieved from https://doi.org/www.ama-assn.org/ama/ama-wire/post/deeper-dive-digital-snake-oil-qa-dr-madara.
Wyatt, J. C., Thimbleby, H., Rastall, P., Hoogewerf, J., Wooldridge, D., & John Williams, J. (2015). What makes a good clinical app? Introducing the RCP health informatics unit checklist. Clinical Medicine, 15(6), 519–521. doi:https://doi.org/10.7861/clinmedicine.15-6-519.
Xu, W., & Liu, L. (2015). mHealth apps: a repository and database of mobile health apps. JMIR mHealth and uHealth, 3(1), e28. doi:https://doi.org/10.2196/mhealth.4026.
Yang, Y. T., & Silverman, R. D. (2014). Mobile health applications: the patchwork of legal and liability issues suggests strategies to improve oversight. Health Affairs, 33(2), 222–227. doi:https://doi.org/10.1377/hlthaff.2013.0958.
Zaidan, S., & Roehrer, E. (2016). Popular mobile phone apps for diet and weight loss: a content analysis. JMIR mHealth and uHealth, 4(3), e80. doi:https://doi.org/10.2196/mhealth.5406.
Zeng, M. L. (2005). Construction of controlled vocabularies: a primer. Retrieved from https://doi.org/marciazeng.slis.kent.edu/Z3919/index.htm.
Author information
Authors and Affiliations
Corresponding author
Additional information
An erratum to this article is available at https://doi.org/10.1007/s41347-017-0021-7.
Rights and permissions
About this article
Cite this article
Maheu, M.M., Nicolucci, V., Pulier, M.L. et al. The Interactive Mobile App Review Toolkit (IMART): a Clinical Practice-Oriented System. J. technol. behav. sci. 1, 3–15 (2016). https://doi.org/10.1007/s41347-016-0005-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41347-016-0005-z