Randomized controlled trial of an automated problem list with improved sensitivity

https://doi.org/10.1016/j.ijmedinf.2007.12.001Get rights and content

Abstract

Purpose

To improve the completeness and timeliness of an electronic problem list, we have developed a system using Natural Language Processing (NLP) to automatically extract potential medical problems from clinical, free-text documents; these problems are then proposed for inclusion in an electronic problem list management application.

Methods

A prospective randomized controlled evaluation of the Automatic Problem List (APL) system in an intensive care unit and in a cardiovascular surgery unit is reported here. A total of 247 patients were enrolled: 76 in an initial control phase and 171 in the randomized controlled trial that followed. During this latter phase, patients were randomly assigned to a control or an intervention group. All patients had their documents analyzed by the system, but the medical problems discovered were only proposed in the problem list for intervention patients. We measured the sensitivity, specificity, positive and negative predictive values, likelihood ratios and the timeliness of the problem lists.

Results

Our system significantly increased the sensitivity of the problem lists in the intensive care unit, from about 9% to 41%, and even 77% if problems automatically proposed but not acknowledged by users were also considered. Timeliness of addition of problems to the list was greatly improved, with a time between a problem's first mention in a clinical document and its addition to the problem list reduced from about 6 days to less than 2 days. No significant effect was observed in the cardiovascular surgery unit.

Introduction

The last decade has witnessed a substantial growth in the amount of medical data recorded for a given patient, along with an increasing pressure to improve the quality of healthcare and reduce medical errors. The problem-oriented, Electronic Health Record (EHR), centered on the problem list, is seen by many as a possible answer to these growing challenges. The problem list is a central place for clinicians to have a concise view of all of a patient's medical problems. The problem list also encourages an orderly process of clinical problem solving, prevents redundant actions [1], and supports the clear documentation of patient condition and clinical decision-making and improves communication among caregivers.

At Intermountain Health Care (IHC), a health maintenance organization serving Utah, the problem list is an important piece of the medical record, and a central component of the clinical information system called HELP2 [2]. To enable its potential benefits, the problem list has to be as accurate, complete and timely as possible. Unfortunately, problem lists are usually incomplete and inaccurate, and are often totally unused. To address this deficiency, we have created an application using Natural Language Processing (NLP) to harvest potential problem list entries from the multiple free-text electronic documents available in a patient's EHR [3], [4]. The medical problems identified are then proposed to the physicians for addition to the official problem list. This system, referred to as the Automatic Problem List (APL) system, is evaluated here. We hypothesize that the use of NLP to automatically provide potential medical problems will improve the completeness, accuracy and timeliness (decreased time between problems identification and their addition to the list) of this Automated Problem List.

Section snippets

Background

The problem list in a Problem-Oriented Medical Record (POMR) was proposed more than three decades ago by Weed [5], [6] as an answer to the complexity of medical knowledge and clinical data, and to address weaknesses in the documentation of medical care. In recent years, the problem-oriented, Computer-based Patient Record (CPR) and the problem list have seen renewed interest as an organizational tool [1], [7], [8], [9], [10], [11], [12], [13]. Advantages to this approach are that the problem

Methods

As mentioned earlier, the Automated Problem List system extracts potential medical problems from free-text medical documents, and uses NLP to achieve this task. The two main components comprising the system are a background application for problem discovery and the problem list management application mentioned above. The background application is responsible for the text processing and analysis and stores extracted problems in the central clinical database. These problems can then be accessed

Results

Ten different reviewers, all physicians, reviewed clinical documents to create the reference standard. Eight were board-certified physicians (most of them in internal medicine), and two were residents with at least 2 years of training. Each reviewer examined an average of 686 different documents with the web-based application described above. The time spent for a review was between 48 and 216 s per document. Reviewers’ overall agreement was very good, with a Finn's R of 0.897 when reviewing

Discussion

This evaluation of our Automated Problem List system suggests that the addition of NLP to improve accuracy and timeliness of the problem list was successful. We measured a significantly increased sensitivity. Clearly, enhancing the problem list management application with NLP made the problem list more complete. We also measured a significantly improved timeliness of the problem list, with an average time difference between a medical problem's first mention in text and its addition to the

Conclusion

The Automated Problem List system that we developed to extract potential medical problems from free-text documents in a patient's EHR has shown satisfying results. This system's goal to improve the problem list's quality by increasing its completeness and timeliness was met, showing higher sensitivity and better timeliness in the intervention group. This was achieved only when the problem list was used. By encouraging the use of a problem list of better quality, this system could potentially

Acknowledgments

This work is supported by a Deseret Foundation Grant (Salt Lake City, Utah). We would like to thank Min Bowman for her help with the modified Problems module. We would also like to thank Greg Gurr for his advices and his help. Scott Narus and Stan Huff also gave us helpful advice and guidance for which we are grateful. Finally, we are especially grateful to Terry Clemmer whose enthusiasm for the problem list made this study possible.

Summary points

What was known before this research?

  • The problem

References (34)

  • J.R. Campbell et al.

    A comparison of four schemes for codification of problem lists

    Proc. Annu. Symp. Comput. Appl. Med. Care

    (1994)
  • M.S. Donaldson et al.

    Improving the master problem list: a case study in changing clinician behavior

    QRB Qual. Rev. Bull.

    (1985)
  • P.L. Elkin et al.

    Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System

    Proc. AMIA Annu. Fall Symp.

    (1997)
  • H. Goldberg et al.

    An evaluation of UMLS as a controlled terminology for the Problem List Toolkit

    Medinfo

    (1998)
  • J.W. Hales et al.

    Extracting medical knowledge for a coded problem list vocabulary from the UMLS Knowledge Sources

    Proc. AMIA Symp.

    (1998)
  • J. Starmer et al.

    Development of a structured problem list management system at vanderbilt

    Proc. AMIA Annu. Fall Symp.

    (1998)
  • D.W. Simborg et al.

    Information factors affecting problem follow-up in ambulatory care

    Med. Care

    (1976)
  • Cited by (44)

    • Automated problem list generation and physicians perspective from a pilot study

      2017, International Journal of Medical Informatics
      Citation Excerpt :

      Unfortunately, this is haphazardly performed, leading to inaccurate, incomplete and duplicative lists. Efforts to automate the creation of the problem list based on EHR data have been made, but were limited to 80 prespecified problems and utilized techniques that may not scale to all possible problems [10–12]. With the advent of advanced natural language processing (NLP) and machine learning techniques [13–15], it is possible for automatic creation and updating of a patients’ problem list based on the data in the EHR.

    • Effect of implementation of a coded problem list entry subsystem

      2016, Computer Methods and Programs in Biomedicine
      Citation Excerpt :

      Limited research has been conducted on this topic. A clinical alerting system [15] and natural language processing [16] have been used for improving the completeness of problem lists. The support systems efficiently improve the completeness of problem lists; however, they can also be a source of tremendous clinical documentation errors, which may affect patient care [17,18].

    • Predicting delirium: A review of risk-stratification models

      2015, General Hospital Psychiatry
      Citation Excerpt :

      Data extraction from electronic medical records requires less labor on the back end, but ultimately depends on reliable assessment and data input from clinical staff, which may not consistently occur. Electronic problems lists, for instance, are frequently inaccurate and incomplete [32]. In addition, the paucity of replication studies for these models, with the exception of Rudolph [25] developing an electronic version of Inouye's 1993 model, is an important limitation of this literature.

    • A trial of indication based prescribing of antihypertensive medications during computerized order entry to improve problem list documentation

      2013, International Journal of Medical Informatics
      Citation Excerpt :

      For example, a study of the Veteran's Administrations EMR [11] found that only 49% of patients with hypertension had the diagnosis on the problem list. Further, a study from Intermountain Healthcare reported that their problem lists are “usually incomplete and inaccurate, and are often totally unused” [12]. The completeness and accuracy of electronic problem lists can be improved using clinical decision support (CDS) to semi-automate the creation of problem list entries via indication based prescribing [13,14], or, as recently reported by Wright, using pop-up screens independent from orders [15].

    • I<inf>KS</inf> index: A knowledge-model driven index to estimate the capability of medical diagnosis systems to produce results

      2013, Expert Systems with Applications
      Citation Excerpt :

      This search led to a pruned ANN architecture which minimized the number of diagnostic factors used during the training phase and, therefore, also decreased the number of nodes in the ANN input and hidden layer, as well as in the Mean Square Error of the trained ANN at the testing phase. In Meystre and Haug (2008) an Automated Problem List system using Natural Language Processing (NLP) was developed to extract potential medical problems from free-text documents in a patient’s Electronic Health Record (EHR). The system significantly increased the sensitivity of the problem lists in the intensive care unit, from about 9–41%, and even 77% if problems were automatically proposed.

    View all citing articles on Scopus
    View full text