Skip to main content
SpringerLink
Log in

Machine learning to predict in-hospital cardiac arrest from patients presenting to the emergency department

  • EM - ORIGINAL
  • Published:
Internal and Emergency Medicine Aims and scope Submit manuscript

Abstract

In-hospital cardiac arrest (IHCA) in the emergency department (ED) is not uncommon but often fatal. Using the machine learning (ML) approach, we sought to predict ED-based IHCA (EDCA) in patients presenting to the ED based on triage data. We retrieved 733,398 ED records from a tertiary teaching hospital over a 7 year period (Jan. 1, 2009–Dec. 31, 2015). We included only adult patients (≥ 18 y) and excluded cases presenting as out-of-hospital cardiac arrest. Primary outcome (EDCA) was identified via a resuscitation code. Patient demographics, triage data, and structured chief complaints (CCs), were extracted. Stratified split was used to divide the dataset into the training and testing cohort at a 3-to-1 ratio. Three supervised ML models were trained and performances were evaluated and compared to the National Early Warning Score 2 (NEWS2) and logistic regression (LR) model by the area under the receiver operating characteristic curve (AUC). We included 316,465 adult ED records for analysis. Of them, 636 (0.2%) developed EDCA. Of the constructed ML models, Random Forest outperformed the others with the best AUC result (0.931, 95% CI 0.911–0.949), followed by Gradient Boosting (0.930, 95% CI 0.909–0.948) and Extra Trees classifier (0.915, 95% CI 0.892–0.936). Although the differences between each of ML models and LR (AUC: 0.905, 95% CI 0.882–0.926) were not significant, all constructed ML models performed significantly better than using the NEWS2 scoring system (AUC 0.678, 95% CI 0.635–0.722). Our ML models showed excellent discriminatory performance to identify EDCA based only on the triage information. This ML approach has the potential to reduce unexpected resuscitation events if successfully implemented in the ED information system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

Not available due to institutional restrictions.

Abbreviations

IHCA:

In-hospital cardiac arrest

OHCA:

Out-of-hospital cardiac arrest

ED:

Emergency department

EMR:

Electronic medical record

EWS:

Early warning score

EDCA:

ED-based IHCA

ML:

Machine learning

NEWS:

National early warning score

CC:

Chief complaint

iMD:

Integrated medical database

NTUH:

National Taiwan University Hospital

TTAS:

Taiwan triage and acuity scale

EMS:

Emergency medical services

GCS:

Glasgow coma scale

SpO2:

Oxygen saturation

SD:

Standard deviation

RF:

Random forest

GB:

Gradient boosting

ET:

Extra trees

AUC:

Area under the receiver operating characteristic curve

AUPRC:

Area under the precision recall curve

NPV:

Negative predictive value

PPV:

Positive predictive value

SMOTE:

Synthetic minority oversampling technique

ESI:

Emergency severity index

CTAS:

Canadian emergency department triage and acuity scale

MTS:

Manchester triage system

References

  1. Andersen LW, Holmberg MJ, Berg KM, Donnino MW, Granfeldt A (2019) In-hospital cardiac arrest: a review. JAMA 321(12):1200–1210. https://doi.org/10.1001/jama.2019.1696

    Article  PubMed  PubMed Central  Google Scholar 

  2. Cummins RO, Chamberlain D, Hazinski MF, Nadkarni V, Kloeck W, Kramer E, Becker L et al (1997) Recommended guidelines for reviewing, reporting, and conducting research on in-hospital resuscitation: the in-hospital “Utstein style’. Am Heart Assoc Ann Emerg Med 29:650–679. https://doi.org/10.1016/s0196-0644(97)70256-7

    Article  CAS  Google Scholar 

  3. Jacobs I, Nadkarni V, Bahr J, Berg RA, Billi JE, Bossaert L et al (2004) Cardiac arrest and cardiopulmonary resuscitation outcome reports: update and simplification of the Utstein templates for resuscitation registries: a statement for healthcare professionals from a task force of the International Liaison Committee on Resuscitation (American Heart Association, European Resuscitation Council, Australian Resuscitation Council, New Zealand Resuscitation Council, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Councils of Southern Africa). Circulation 110:3385–3397. https://doi.org/10.1161/01.CIR.0000147236.85306.15

    Article  PubMed  Google Scholar 

  4. Yan S, Gan Y, Jiang N, Wang R, Chen Y, Luo Z et al (2020) The global survival rate among adult out-of-hospital cardiac arrest patients who received cardiopulmonary resuscitation: a systematic review and meta-analysis. Crit Care 24:61. https://doi.org/10.1186/s13054-020-2773-2

    Article  PubMed  PubMed Central  Google Scholar 

  5. Kumar G, Nanchal R (2013) Trends in survival after in-hospital cardiac arrest. N Engl J Med 14(368):680. https://doi.org/10.1056/NEJMc1215155

    Article  CAS  Google Scholar 

  6. Camargo CA Jr, Tsai CL, Sullivan AF, Cleary PD, Gordon JA, Guadagnoli E et al (2012) Safety climate and medical errors in 62 US emergency departments. Ann Emerg Med 60:555-563.e20. https://doi.org/10.1016/j.annemergmed.2012.02.018

    Article  PubMed  Google Scholar 

  7. Horwitz LI, Green J, Bradley EH (2010) US emergency department performance on wait time and length of visit. Ann Emerg Med 55:133–141. https://doi.org/10.1016/j.annemergmed.2009.07.023

    Article  PubMed  Google Scholar 

  8. Churpek MM, Yuen TC, Huber MT, Park SY, Hall JB, Edelson DP (2012) Predicting cardiac arrest on the wards: a nested case-control study. Chest 141:1170–1176. https://doi.org/10.1378/chest.11-1301

    Article  PubMed  Google Scholar 

  9. Gerry S, Bonnici T, Birks J, Kirtley S, Virdee PS, Watkinson PJ et al (2020) Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ 369:m1501. https://doi.org/10.1136/bmj.m1501

    Article  PubMed  PubMed Central  Google Scholar 

  10. Goecks J, Jalili V, Heiser LM, Gray JW (2020) How machine learning will transform biomedicine. Cell 181:92–101. https://doi.org/10.1016/j.cell.2020.03.022

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Rajkomar A, Dean J, Kohane I (2019) Machine Learning in Medicine. Reply N Engl J Med 27(380):2589–2590. https://doi.org/10.1056/NEJMc1906060

    Article  Google Scholar 

  12. Chang HK, Wu CT, Liu JH, Lim WS, Wang HC, Chiu SI et al (2019) Early Detecting In-Hospital Cardiac Arrest Based on Machine Learning on Imbalanced Data, 2019 IEEE International Conference on Healthcare Informatics (ICHI), pp. 1–10. https://doi.org/10.1109/ICHI.2019.8904504.

  13. Royal College of Physicians National Early Warning Score (NEWS): Standardising the assessment of acute illness severity in the NHS. Report of a working party. London: RCP, 2012 https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-news-2. Accessed 3 May 2022

  14. Ng CJ, Yen ZS, Tsai JC, Chen LC, Lin SJ, Sang YY et al (2011) Validation of the Taiwan triage and acuity scale: a new computerised five-level triage system. Emerg Med J 28:1026–1031. https://doi.org/10.1136/emj.2010.094185

    Article  PubMed  Google Scholar 

  15. Nolan JP, Berg RA, Andersen LW, Bhanji F, Chan PS, Donnino MW et al (2019) Cardiac arrest and cardiopulmonary resuscitation outcome reports: update of the utstein resuscitation registry template for in-hospital cardiac arrest: a consensus report from a task force of the international liaison committee on resuscitation (American Heart Association, European Resuscitation Council, Australian and New Zealand Council on Resuscitation, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Council of Southern Africa, Resuscitation Council of Asia). Resuscitation 144:166–177. https://doi.org/10.1016/j.resuscitation.2019.08.021

    Article  PubMed  Google Scholar 

  16. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Art Intell Res 16:321–357. https://doi.org/10.1613/jair.953

    Article  Google Scholar 

  17. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845

    Article  CAS  PubMed  Google Scholar 

  18. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O et al (2011) Scikit-learn: machine learning in python. JMLR 12:2825–2830

    Google Scholar 

  19. S. Džeroski P, Panov B, Ženko, (2009) Ensemble methods in machine learning. In: Meyer T (ed) Encyclopedia of Complexity and Systems Science. Springer, New York, pp 5317–5325

    Google Scholar 

  20. Trevethan R (2017) Sensitivity, specificity, and predictive values: foundations, pliabilities, and pitfalls in research and practice. Front Public Health 5:307. https://doi.org/10.3389/fpubh.2017.00307

    Article  PubMed  PubMed Central  Google Scholar 

  21. Ranganathan P, Aggarwal R (2018) Common pitfalls in statistical analysis: understanding the properties of diagnostic tests-part 1. Perspect Clin Res 9:40–43. https://doi.org/10.4103/picr.PICR_170_17

    Article  PubMed  PubMed Central  Google Scholar 

  22. Peberdy MA, Kaye W, Ornato JP, Larkin GL, Nadkarni V, Mancini ME et al (2003) Cardiopulmonary resuscitation of adults in the hospital: a report of 14720 cardiac arrests from the national registry of cardiopulmonary resuscitation. Resuscitation 58:297–308. https://doi.org/10.1016/s0300-9572(03)00215-6

    Article  PubMed  Google Scholar 

  23. Shih CL, Lu TC, Jerng JS, Lin CC, Liu YP, Chen WJ et al (2007) A web-based Utstein style registry system of in-hospital cardiopulmonary resuscitation in Taiwan. Resuscitation 72:394–403. https://doi.org/10.1016/j.resuscitation.2006.07.020

    Article  PubMed  Google Scholar 

  24. Høybye M, Stankovic N, Holmberg M, Christensen HC, Granfeldt A, Andersen LW (2021) In-hospital vs. out-of-hospital cardiac arrest: patient characteristics and survival. Resuscitation 158:157–165. https://doi.org/10.1016/j.resuscitation.2020.11.016

    Article  PubMed  Google Scholar 

  25. Christ M, Grossmann F, Winter D, Bingisser R, Platz E (2010) Modern triage in the emergency department. Dtsch Arztebl Int 107:892–898. https://doi.org/10.3238/arztebl.2010.0892

    Article  PubMed  PubMed Central  Google Scholar 

  26. van der Wulp I, Schrijvers AJ, van Stel HF (2009) Predicting admission and mortality with the emergency severity index and the manchester triage system: a retrospective observational study. Emerg Med J 26:506–509. https://doi.org/10.1136/emj.2008.063768

    Article  PubMed  Google Scholar 

  27. Martins HM, Cuña LM, Freitas P (2009) Is manchester (MTS) more than a triage system? A study of its association with mortality and admission to a large Portuguese hospital. Emerg Med J 26:183–186. https://doi.org/10.1136/emj.2008.060780

    Article  CAS  PubMed  Google Scholar 

  28. Hinson JS, Martinez DA, Schmitz PSK, Toerper M, Radu D, Scheulen J et al (2018) Accuracy of emergency department triage using the emergency severity index and independent predictors of under-triage and over-triage in Brazil: a retrospective cohort analysis. Int J Emerg Med 11:3. https://doi.org/10.1186/s12245-017-0161-8

    Article  PubMed  PubMed Central  Google Scholar 

  29. Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D (2016) An electronic emergency triage system to improve patient distribution by critical outcomes. J Emerg Med 50:910–918. https://doi.org/10.1016/j.jemermed.2016.02.026

    Article  PubMed  Google Scholar 

  30. Mitchell OJL, Edelson DP, Abella BS (2020) Predicting cardiac arrest in the emergency department. J Am Coll Emerg Physicians Open 1:321–326. https://doi.org/10.1002/emp2.12015

    Article  PubMed  PubMed Central  Google Scholar 

  31. Green M, Lander H, Snyder A, Hudson P, Churpek M, Edelson D (2018) Comparison of the between the flags calling criteria to the mews, news and the electronic cardiac arrest risk triage (eCART) score for the identification of deteriorating ward patients. Resuscitation 123:86–91. https://doi.org/10.1016/j.resuscitation.2017.10.028

    Article  PubMed  Google Scholar 

  32. Ong ME, Lee Ng CH, Goh K, Liu N, Koh ZX, Shahidah N et al (2012) Prediction of cardiac arrest in critically ill patients presenting to the emergency department using a machine learning score incorporating heart rate variability compared with the modified early warning score. Crit Care 16:R108. https://doi.org/10.1186/cc11396

    Article  PubMed  PubMed Central  Google Scholar 

  33. Chae M, Han S, Gil H, Cho N, Lee H (2021) Prediction of in-hospital cardiac arrest using shallow and deep learning. Diagnostics (Basel) 11(7):1255. https://doi.org/10.3390/diagnostics11071255

    Article  PubMed  Google Scholar 

  34. Kwon JM, Lee Y, Lee Y, Lee S, Park J (2018) An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc 7:e008678. https://doi.org/10.1161/JAHA.118.008678

    Article  PubMed  PubMed Central  Google Scholar 

  35. Kohl M (2012) Performance Measures in Binary Classification. Int J Stat Med Res 1:79–81. https://doi.org/10.6000/1929-6029.2012.01.01.08

    Article  Google Scholar 

  36. Saito T, Rehmsmeier M (2015) The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10(3):e0118432. https://doi.org/10.1371/journal.pone.0118432

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Alghamdi M, Al-Mallah M, Keteyian S, Brawner C, Ehrman J, Sakr S (2017) Predicting diabetes mellitus using SMOTE and ensemble machine learning approach: The Henry Ford ExercIse Testing (FIT) project. PLoS ONE 12:e0179805. https://doi.org/10.1371/journal.pone.0179805

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Fotouhi S, Asadi S, Kattan MW (2019) A comprehensive data level analysis for cancer diagnosis on imbalanced data. J Biomed Inform 90:103089. https://doi.org/10.1016/j.jbi.2018.12.003

    Article  PubMed  Google Scholar 

  39. Lee YW, Choi JW, Shin EH (2021) Machine learning model for predicting malaria using clinical information. Comput Biol Med 129:104151. https://doi.org/10.1016/j.compbiomed.2020.104151

    Article  PubMed  Google Scholar 

  40. Sung CW, Lu TC, Wang CH, Chou EH, Ko CH, Huang CH et al (2022) In-hospital cardiac arrest in united states emergency departments, 2010–2018. Front Cardiovasc Med 9:874461. https://doi.org/10.3389/fcvm.2022.874461

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors would like to express their thanks to the staff of Department of Medical Research for providing clinical data from National Taiwan University Hospital-integrated Medical Database (NTUH-iMD).

Funding

This work was supported by the Ministry of Science and Technology Taiwan [111-2634-F-002 -015]; and the National Taiwan University Hospital [105-N3102]. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. All of the authors report no conflict of interest.

Author information

Authors and Affiliations

  1. Department of Emergency Medicine, National Taiwan University Hospital, 7, Zhongshan S. Rd, Taipei City, 100, Taiwan

    Tsung-Chien Lu, Chih-Hung Wang, Fan-Ya Chou, Edward Pei-Chuan Huang, Chu-Lin Tsai, Matthew Huei-Ming Ma, Cheng-Chung Fang & Chien-Hua Huang

  2. Department of Emergency Medicine, College of Medicine, National Taiwan University, Taipei, Taiwan

    Tsung-Chien Lu, Chih-Hung Wang, Edward Pei-Chuan Huang, Chu-Lin Tsai, Matthew Huei-Ming Ma, Cheng-Chung Fang & Chien-Hua Huang

  3. Department of Emergency Medicine, Far Eastern Memorial Hospital, New Taipei City, Taiwan

    Jen-Tang Sun

  4. Department of Emergency Medicine, Baylor Scott and White All Saints Medical Center, Fort Worth, TX, USA

    Eric H. Chou

  5. Department of Emergency Medicine, National Taiwan University Hsinchu Branch, Hsinchu, Taiwan

    Edward Pei-Chuan Huang

  6. Department of Emergency Medicine, National Taiwan University Yunlin Branch, Yunlin, Taiwan

    Matthew Huei-Ming Ma

Authors
  1. Tsung-Chien Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chih-Hung Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fan-Ya Chou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jen-Tang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eric H. Chou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Edward Pei-Chuan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chu-Lin Tsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Matthew Huei-Ming Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Cheng-Chung Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Chien-Hua Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conception and design of the study: TCL, CHW, CLT. Acquisition of data: TCL, CLT, CHH. Analysis and interpretation of data: TCL, FYC. Drafting the article: TCL, CHW. Revising it critically for important intellectual content: JTS, EHC, MHMM, CCF. Final approval of the version to be submitted: All authors contributed to the final version. CLT takes responsibility for the paper as a whole.

Corresponding author

Correspondence to Chu-Lin Tsai.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethics approval

This study was approved by the institutional review board of the National Taiwan University Hospital (201606072RINA) and waived the requirement for informed consent.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 1087 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, TC., Wang, CH., Chou, FY. et al. Machine learning to predict in-hospital cardiac arrest from patients presenting to the emergency department. Intern Emerg Med 18, 595–605 (2023). https://doi.org/10.1007/s11739-022-03143-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11739-022-03143-1

Keywords

Search

Navigation