Leveraging user's performance in reporting patient safety events by utilizing text prediction in narrative data entry

https://doi.org/10.1016/j.cmpb.2016.03.031Get rights and content

Highlights

  • A two-group randomized study of testing the usability of text prediction functions in patient safety event reporting.

  • 52 experienced nurses in a top-level hospital in China participated in the experiment.

  • Text prediction prompted the user's engagement of the narrative comment field.

  • Text prediction improved the efficiency by leveraging the text generation rate.

  • Text prediction ameliorated the data completeness.

Abstract

Background

Narrative data entry pervades computerized health information systems and serves as a key component in collecting patient-related information in electronic health records and patient safety event reporting systems. The quality and efficiency of clinical data entry are critical in arriving at an optimal diagnosis and treatment. The application of text prediction holds potential for enhancing human performance of data entry in reporting patient safety events.

Objective

This study examined two functions of text prediction intended for increasing efficiency and data quality of text data entry reporting patient safety events.

Methods

The study employed a two-group randomized design with 52 nurses. The nurses were randomly assigned into a treatment group or a control group with a task of reporting five patient fall cases in Chinese using a web-based test system, with or without the prediction functions. T-test, Chi-square and linear regression model were applied to evaluating the outcome differences in free-text data entry between the groups.

Results

While both groups of participants exhibited a good capacity for accomplishing the assigned task of reporting patient falls, the results from the treatment group showed an overall increase of 70.5% in text generation rate, an increase of 34.1% in reporting comprehensiveness score and a reduction of 14.5% in the non-adherence of the comment fields. The treatment group also showed an increasing text generation rate over time, whereas no such an effect was observed in the control group.

Conclusion

As an attempt investigating the effectiveness of text prediction functions in reporting patient safety events, the study findings proved an effective strategy for assisting reporters in generating complementary free text when reporting a patient safety event. The application of the strategy may be effective in other clinical areas when free text entries are required.

Introduction

Many attempts have been made to investigate the difficulties in clinical data entry in order to promote the acceptance and quality-in-use of clinical information systems [1], [2], [3], [4]. With the advance of non-narrative entry templates and natural language processing techniques, the application of structured entries in clinical information systems is increasing because of the merit of interoperability and reuse. However, given the complexity of healthcare, failure to include essential fields and lack of options in structure entries are still common. Structured entries also received critiques due to lack of flexibility and expressiveness in clinical communication. Therefore, unstructured narrative data entry plays an indispensable role in clinical data entry.

For patient safety event reporting, narrative entry has been a prevalent and dominant format. However, it is now in a transition of becoming a supplementary format to non-narrative forms released by national and international organizations [5], [6]. Narrative comment field is intended to collect case details beyond structured entries. Previous studies showed that voluntary reporters usually ignored this field or described events with inaccurate and incomplete terms and sentences [7], [8]. Two major barriers are identified due to the multitasking and busy nature of healthcare and the lack of languages or knowledge for reporters to describe patient safety events in detail [9]. To help remove the barriers and achieve quality-in-use, it requires a user-centered design process which embraces cultural, strategic and technical considerations [10]. We proposed to examine the utilization of text prediction applied in the narrative fields in reporting patient safety events.

Text prediction, also known as word, sentence or context prediction originated in augmentative and alternative communication (AAC) to increase text generation rates for people with the disabilities of motor or speech impairment [11], [12]. The advance of natural language processing techniques has brought text prediction into a broader scope of daily computing activities, such as mobile computing [13] and radiography reports [14]. Studies on optimization [15] and impact [16] of text prediction have received much attention. This study focusing on the impact evaluation, would respond three basic concerns regarding text prediction in reporting patient safety events. These concerns include whether text prediction would increase (1) reporter's engagement in narrative comment field; (2) quality of narrative entry, which is highly valuable in generating actionable knowledge yet received little attention; (3) efficiency of reporting patient safety events, which remains unclear based on mixed results across the fields [16]. In this study, we employed a two-group randomized experiment to examine the impact of text prediction in the three aspects.

Section snippets

Background

This study is grounded in a user-centered design of patient safety event reporting systems. Reporting systems in use show the problems of underreporting [17] and low quality of reports [8], [18], though efforts from all levels are made to improve the systems [19], [20]. Prior to this study, we have consecutively conducted heuristic evaluation, cognitive task analysis and think-aloud user testing [4], [7], [21], [22] which revealed interface representational issues and identified that the

Participants

Potential candidates who were nurses and experienced in reporting and analyzing patient safety events in Tianjin First Central Hospital (TFCH) in Tianjin, China were identified and invited to participate in the study. Two candidates were on leave of absence during the study period, and three candidates felt not confident with operating computers. As a result, the remaining 52 nurses from 21 clinical departments enrolled in the study. All nurses were female between the ages of 30 and 52 years.

Results

All participants, with each reported five cases, successfully completed the experiment and thus generated 260 reports. There were 25 participants allocated in the control and 27 in the in the treatment group, accounting for 125 and 135 reports, respectively. The means of participants' ages were 43.6 ± 5.8 versus 41.1 ± 6.6 years in the control and treatment group. The 260 reports contained 2849 MCQ answers and 238 unstructured narrative comments. The completion times of MCQs were 131.0 ± 50.0

Discussion

Clinicians working under time constraints are usually expected to enter data for documentation in a timely and efficient manner [31], [32]. The comprehensiveness of entered data is critical to clinicians' decision-making and creation of actionable knowledge. To accommodate the expectation in patient safety event reporting, we introduced two text prediction functions of CL and AS, which are attached to narrative comment field pervasively used in patient safety reporting systems. A two-group

Conclusion

Text data entry, as an indispensable format in clinical information systems, was demonstrated to be enhanced by the text prediction functions in terms of efficiency, data quality and engagement in a two-group randomized experiment. This study disclosed the necessity of the text prediction functions in facilitating experienced domain users in text data entry when reporting patient safety events. The quality of event reporting plays a key role in learning from the events. Simply counting numbers

Authors' contributions

Yang Gong: Responsible for study design, data collection and analysis, and coauthored the manuscript with review and revision responsibilities.

Lei Hua: Responsible for the conceptualization, study design, prototype development, testing session observation, data collection and analysis, coauthored the manuscript.

Shen Wang: Organized and coordinated with the study participants, assisted in preparing testing cases, data analysis and had review responsibilities.

Conflicts of interest

We declare there are no conflicts of interest involved in the research.

Summary table

Known on the topic

  • Text data entry is indispensable toward the quality care delivery but suffered from low quality and user's engagement.

  • Text prediction is widely used in mobile computing.

Our contribution

  • Text prediction can increase the efficiency, data quality and user's engagement of text data entry if the functions were properly designed.

  • Text prediction would be still helpful for experienced users to enhance their performance.

  • An initial investigation about the text prediction functions in

Acknowledgments

The authors express heartfelt thanks to Xiao Liu, RN and Ying Sun, RN at the Office of Nursing at Tianjin First Central Hospital in China who made numerous efforts helping review and edit the testing cases, derived proper text predictor responses, scheduled the testing sessions, and graded the subjects' commentary data. This project was in part supported by a grant on patient safety from the University of Texas System (grant number #156374) and a grant from AHRQ (grant number 1R01HS022895).

References (39)

  • J. Kim et al.

    Results of a survey on medical error reporting systems in Korean hospitals

    Int. J. Med. Inform

    (2006)
  • J.P. Matts et al.

    Properties of permuted-block randomization in clinical trials

    Control. Clin. Trials

    (1988)
  • L. Poissant et al.

    The impact of electronic health records on time efficiency of physicians and nurses: a systematic review

    J. Am. Med. Inform. Assoc

    (2005)
  • C.J. McDonald

    The barriers to electronic medical record systems and how to overcome them

    J. Am. Med. Inform. Assoc

    (1997)
  • B. Kaplan

    Reducing barriers to physician data entry for computer-based patient records

    Top. Health Inf. Manage

    (1994)
  • S.H. Walsh

    The clinician's perspective on electronic health records and how they can affect patient care

    BMJ

    (2004)
  • L. Hua et al.

    Text prediction on structured data entry in healthcare: a two-group randomized usability study measuring the prediction impact on user performance

    Appl. Clin. Inform

    (2014)
  • AHRQ

    Users' Guide AHRQ Common Formats for Patient Safety Organizations

    (2008)
  • Towards an international classification for patient safety: the conceptual framework

  • L. Hua et al.

    Design of a user-centered voluntary patient safety reporting system: understanding the time and response variances by retrospective think-aloud protocols

    Stud. Health Technol. Inform

    (2013)
  • Y. Gong

    Data consistency in a voluntary medical incident reporting system

    J. Med. Syst

    (2009)
  • R.J. Holden et al.

    A review of medical error reporting system design considerations and a proposed cross-level systems research framework

    Hum. Factors

    (2007)
  • G. Klein

    Sources of Power: How People Make Decisions

    (1999)
  • D. Beukelman et al.

    Augmentative and alternative communication

    (2005)
  • N. Garay-Vitoria et al.

    Text prediction systems: a survey

    Univ. Access Inf. Soc

    (2006)
  • S.I. Mackenzie et al.

    Text entry for mobile computing: models and methods, theory and practice

    Human-Computer Interaction

    (2002)
  • J. Eng et al.

    Informatics in radiology (infoRAD): radiology report entry with automatic phrase completion driven by language modeling

    Radiographics

    (2004)
  • S. Hunnicutt et al.

    Improving word prediction using Markov models and Heuristic methods

    Augment. Altern. Commun

    (2001)
  • H.H. Koester et al.

    Learning and performance of able-bodied individuals using scanning systems with and without word prediction

    Assist. Technol

    (1994)
  • Cited by (6)

    View full text