Comparing a deep learning model’s diagnostic performance to that of radiologists to detect Covid -19 features on chest radiographs

 

 Permissions and Reprints

Abstract

Background: Whether the sensitivity of Deep Learning (DL) models to screen chest radiographs (CXR) for CoVID-19 can approximate that of radiologists, so that they can be adopted and used if real-time review of CXRs by radiologists is not possible, has not been explored before. Objective: To evaluate the diagnostic performance of a doctor-trained DL model (Svita_DL8) to screen for COVID-19 on CXR, and to compare the performance of the DL model with that of expert radiologists. Materials and Methods: We used a pre-trained convolutional neural network to develop a publicly available online DL model to evaluate CXR examinations saved in .jpeg or .png format. The initial model was subsequently curated and trained by an internist and a radiologist using 1062 chest radiographs to classify a submitted CXR as either normal, COVID-19, or a non-COVID-19 abnormal. For validation, we collected a separate set of 430 CXR examinations from numerous publicly available datasets from 10 different countries, case presentations, and two hospital repositories. These examinations were assessed for COVID-19 by the DL model and by two independent radiologists. Diagnostic performance was compared between the model and the radiologists and the correlation coefficient calculated. Results: For detecting COVID-19 on CXR, our DL model demonstrated sensitivity of 91.5%, specificity of 55.3%, PPV 60.9%, NPV 77.9%, accuracy 70.1%, and AUC 0.73 (95% CI: 0.86, 0.95). There was a significant correlation (r = 0.617, P = 0.000) between the results of the DL model and the radiologists’ interpretations. The sensitivity of the radiologists is 96% and their overall diagnostic accuracy is 90% in this study. Conclusions: The DL model demonstrated high sensitivity for detecting COVID-19 on CXR. Clinical Impact: The doctor trained DL tool Svita_DL8 can be used in resource-constrained settings to quickly triage patients with suspected COVID-19 for further in-depth review and testing.

Publication History

Received: 27 November 2020

Accepted: 17 December 2020

Article published online:
13 July 2021

© 2021. Indian Radiological Association. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial-License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Thieme Medical and Scientific Publishers Private Ltd.
A-12, Second Floor, Sector -2, NOIDA -201301, India

• References

• 1 Fang Y, Zhang H, Xie J, Lin M, Ying L, Pang P. et al. Sensitivity of chest CT for COVID-19: Comparison to RT-PCR. Radiology 2020; 296: E115-7
  CrossrefPubMedGoogle Scholar
• 2 Simpson S, Kay FU, Abbara S, Bhalla S, Chung JH, Chung M. et al. Radiological society of North America expert consensus statement on reporting chest CT findings related to COVID-19: Endorsed by the Society of thoracic radiology, the American college of radiology, and RSNA. Radiology: J Thorac Imaging 2020; 2: e200152
  Google Scholar
• 3 Li Y, Xia L. Coronavirus disease 2019 (COVID-19): Role of chest CT in diagnosis and management. AJR Am J Roentgenol 2020; 214: 1280-6
  CrossrefPubMedGoogle Scholar
• 4 Jacobi A, Chung M, Bernheim A, Eber C. Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review. Clin Imaging 2020; 64: 35-42
  CrossrefPubMedGoogle Scholar
• 5 Wong HYF, Lam HYS, Fong AH-T, Leung ST, Chin TW-Y, Lo CSY. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 2020; 296: E72-8
  CrossrefPubMedGoogle Scholar
• 6 ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection [Internet]. Acr.org. 2020. Available from: https://www.acr.org/Advocacy-and-Economics/ACR-Position-Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection [Last cited on 2020 May 07].
• 7 Nolan TM, Oberklaid F, Boldt D. Radiological services in a hospital emergency department—An evaluation of service delivery and radiograph interpretation. J Paediatr Child Health 1984; 20: 109-12
  CrossrefPubMedGoogle Scholar
• 8 Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S. et al. ImageNet large scale visual recognition challenge. Int J Comput Vis 2015; 115: 211-52
  CrossrefGoogle Scholar
• 9 Qin ZZ, Sander MS, Rai B, Titahong CN, Sudrungrot S, Laah SN. et al. Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy of three deep learning systems. Sci Rep 2019; 9: 15000
  CrossrefPubMedGoogle Scholar
• 10 Li L, Qin L, Xu Z, Yin Y, Wang X, Kong B. et al. Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT. Radiology 2020:200905. Published online 2020 Mar 19. doi: 10.1148/radiol.2020200905
• 11 Oh Y, Park S, Ye JC. Deep learning Covid-19 features on CXR using limited training data sets. IEEE Trans Med Imaging 2020; 39: 2688-700
  CrossrefPubMedGoogle Scholar
• 12 Rajaraman S, Antani S. Weakly labeled data augmentation for deep learning: A study on COVID-19 detection in chest X-rays. Diagnostics 2020; 10: 358
  CrossrefPubMedGoogle Scholar
• 13 Abbas A, Abdelsamea MM, Gaber MM. Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network. Appl Intell 2020. https://doi.org/10.1007/s10489-020-01829-7.
• 14 Hall LO, Paul R, Goldgof DB, Goldgof GM. Finding covid-19 from chest x-rays using deep learning on a small dataset. arXiv preprintarXiv: 2004.02060. 2020.arXiv: 2004.02060
• 15 Murphy K, Smits H, Knoops AJG, Korst MBJM, Samson T, Scholten ET. et al. COVID-19 on the chest radiograph: A multi-reader evaluation of an AI system. Radiology 2020; 296: E166-72
  CrossrefPubMedGoogle Scholar
• 16 Chau T-N, Lee P-O, Choi K-W, Lee C-M, Ma K-F, Tsang T-Y. et al. Value of initial chest radiographs for predicting clinical outcomes in patients with severe acute respiratory syndrome. Am J Med 2004; 117: 249-54
  CrossrefPubMedGoogle Scholar
• 17 Timeline of WHO’s response to COVID-19 [Internet]. Who.int. 2020. Available from: https://www.who.int/news-room/detail/29-06-2020-covidtimeline [Last cited on 2020 Jun 29].
• 18 Force AD, Ranieri VM, Rubenfeld GD, Thompson BT, Ferguson ND, Caldwell E. Acute respiratory distress syndrome. JAMA 2012; 307: 2526-33
  PubMedGoogle Scholar
• 19 Gibson PG, Qin L, Puah S. COVID-19 acute respiratory distress syndrome (ARDS): Clinical features and differences from typical pre-COVID-19 ARDS. Med J Aust 2020; 213: 54-56 e1.
  CrossrefPubMedGoogle Scholar

• Supplementary Material