Skip to main content
SpringerLink
Log in

Post-imaging pulmonary nodule mathematical prediction models: are they clinically relevant?

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

Post-imaging mathematical prediction models (MPMs) provide guidance for the management of solid pulmonary nodules by providing a lung cancer risk score from demographic and radiologists-indicated imaging characteristics. We hypothesized calibrating the MPM risk score threshold to a local study cohort would result in improved performance over the original recommended MPM thresholds. We compared the pre- and post-calibration performance of four MPM models and determined if improvement in MPM prediction occurs as nodules are imaged longitudinally.

Materials and methods

A common cohort of 317 individuals with computed tomography-detected, solid nodules (80 malignant, 237 benign) were used to evaluate the MPM performance. We created a web-based application for this study that allows others to easily calibrate thresholds and analyze the performance of MPMs on their local cohort. Thirty patients with repeated imaging were tested for improved performance longitudinally.

Results

Using calibrated thresholds, Mayo Clinic and Brock University (BU) MPMs performed the best (AUC = 0.63, 0.61) compared to the Veteran’s Affairs (0.51) and Peking University (0.55). Only BU had consensus with the original MPM threshold; the other calibrated thresholds improved MPM accuracy. No significant improvements in accuracy were found longitudinally between time points.

Conclusions

Calibration to a common cohort can select the best-performing MPM for your institution. Without calibration, BU has the most stable performance in solid nodules ≥ 8 mm but has only moderate potential to refine subjects into appropriate workup. Application of MPM is recommended only at initial evaluation as no increase in accuracy was achieved over time.

Key Points

• Post-imaging lung cancer risk mathematical predication models (MPMs) perform poorly on local populations without calibration.

• An application is provided to facilitate calibration to new study cohorts: the Mayo Clinic model, the U.S. Department of Veteran’s Affairs model, the Brock University model, and the Peking University model.

• No significant improvement in risk prediction occurred in nodules with repeated imaging sessions, indicating the potential value of risk prediction application is limited to the initial evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Abbreviations

AUC:

Area under the curve

BTS:

British Thoracic Society

BU:

Brock University model

COPD:

Chronic obstructive pulmonary disorder

CT:

Computed tomography

MAD:

Median absolute deviation

MC:

Mayo Clinic model

MPMs :

Mathematical prediction models

PR:

Precision recall

PU:

Peking University model

ROC:

Receiver-operator characteristic

TP_1:

Initial imaging encounter on which the pulmonary nodule was identified

TP_F:

Final imaging encounter before pulmonary nodule diagnosis

VA:

U.S. Department of Veterans Affairs model

References

  1. Siegel RL, Miller KD, Jemal A (2017) Cancer statistics, 2017. CA Cancer J Clin 67:7–30

    Article  PubMed  Google Scholar 

  2. MacMahon H, Naidich DP, Goo JM et al (2017) Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017. Radiology 284:228–243

    Article  PubMed  Google Scholar 

  3. American College of Radiology (2014). Lung CT screening reporting and data system (Lung-RADS). Available via: https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Lung-Rads. Accessed 12/12/2018

  4. Gould MK, Donington J, Lynch WR et al (2013) Evaluation of individuals with pulmonary nodules: when is it lung cancer? Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 143:e93S–e120S

    Article  PubMed  PubMed Central  Google Scholar 

  5. Mehta HJ, Mohammed TL, Jantz MA (2017) The American College of Radiology Lung Imaging Reporting and Data System: potential drawbacks and need for revision. Chest 151:539–543

    Article  PubMed  Google Scholar 

  6. Pinsky PF, Gierada DS, Black W et al (2015) Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med 162:485–491

    Article  PubMed  PubMed Central  Google Scholar 

  7. van Riel SJ, Ciompi F, Jacobs C et al (2017) Malignancy risk estimation of screen-detected nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN guidelines. Eur Radiol 27:4019–4029

    Article  PubMed  PubMed Central  Google Scholar 

  8. Gray EP, Teare MD, Stevens J, Archer R (2016) Risk prediction models for lung cancer: a systematic review. Clin Lung Cancer 17:95–106

    Article  PubMed  Google Scholar 

  9. Swensen SJ, Silverstein MD, Ilstrup DM, Schleck CD, Edell ES (1997) The probability of malignancy in solitary pulmonary nodules. Application to small radiologically indeterminate nodules. Arch Intern Med 157:849–855

    Article  CAS  PubMed  Google Scholar 

  10. Gould MK, Ananth L, Barnett PG, Veterans Affairs SNAP Cooperative Study Group (2007) A clinical model to estimate the pretest probability of lung cancer in patients with solitary pulmonary nodules. Chest 131:383–388

  11. McWilliams A, Tammemagi MC, Mayo JR et al (2013) Probability of cancer in pulmonary nodules detected on first screening CT. N Engl J Med 369:910–919

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Li Y, Wang J (2012) A mathematical model for predicting malignancy of solitary pulmonary nodules. World J Surg 36:830–835

    Article  PubMed  Google Scholar 

  13. Herder GJ, van Tinteren H, Golding RP et al (2005) Clinical prediction model to characterize pulmonary nodules: validation and added value of 18F-fluorodeoxyglucose positron emission tomography. Chest 128:2490–2496

    Article  PubMed  Google Scholar 

  14. Baldwin DR, Callister ME (2015) The British Thoracic Society guidelines on the investigation and management of pulmonary nodules. Thorax 70:794–798

    Article  PubMed  Google Scholar 

  15. Hammer MM, Nachiappan AC, Barbosa EJM Jr (2018) Limited utility of pulmonary nodule risk calculators for managing large nodules. Curr Probl Diagn Radiol 47:23–27

  16. Al-Ameri A, Malhotra P, Thygesen H et al (2015) Risk of malignancy in pulmonary nodules: a validation study of four prediction models. Lung Cancer 89:27–30

    Article  PubMed  Google Scholar 

  17. Mehta HJ, Ravenel JG, Shaftman SR et al (2014) The utility of nodule volume in the context of malignancy prediction for small pulmonary nodules. Chest 145:464–472

    Article  PubMed  Google Scholar 

  18. Perandini S, Soardi GA, Motton M, Montemezzi S (2015) Critique of Al-Ameri et al. (2015) - risk of malignancy in pulmonary nodules: a validation study of four prediction models. Lung Cancer 90:118–119

    Article  PubMed  Google Scholar 

  19. Regan EA, Hokanson JE, Murphy JR et al (2010) Genetic epidemiology of COPD (COPDGene) study design. COPD 7:32–43

    Article  PubMed  Google Scholar 

  20. Schwartz AG, Lusk CM, Wenzlaff AS et al (2016) Risk of lung cancer associated with COPD phenotype based on quantitative image analysis. Cancer Epidemiol Biomarkers Prevv 25:1341–1347

  21. Swensen SJ, Silverstein MD, Edell ES et al (1999) Solitary pulmonary nodules: clinical prediction model versus physicians. Mayo Clin Proc 74:319–329

    Article  CAS  PubMed  Google Scholar 

  22. Cummings SR, Lillington GA, Richard RJ (1986) Managing solitary pulmonary nodules. The choice of strategy is a “close call”. Am Rev Respir Dis 134:453–460

    CAS  PubMed  Google Scholar 

  23. Chung K, Mets OM, Gerke PK et al (2018) Brock malignancy risk calculator for pulmonary nodules: validation outside a lung cancer screening population. Thorax 73:857–863

    Article  PubMed  Google Scholar 

  24. Youden WJ (1950) Index for rating diagnostic tests. Cancer 3:32–35

    Article  CAS  PubMed  Google Scholar 

  25. Jesse D, Goadrich M (2006) The relationship between precision-recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, 25–29 June 2006, pp 233–240

  26. McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–157

    Article  CAS  PubMed  Google Scholar 

  27. Callister ME, Baldwin DR, Akram AR et al (2015) British Thoracic Society guidelines for the investigation and management of pulmonary nodules. Thorax 70(Suppl 2):ii1–ii54

    Article  PubMed  Google Scholar 

  28. McNitt-Gray MF, Kim GH, Zhao B et al (2015) Determining the variability of lesion size measurements from CT patient data sets acquired under “no change” conditions. Transl Oncol 8:55–64

    Article  PubMed  PubMed Central  Google Scholar 

  29. Lin H, Huang C, Wang W, Luo J, Yang X, Liu Y (2017) Measuring interobserver disagreement in rating diagnostic characteristics of pulmonary nodule using the lung imaging database consortium and image database resource initiative. Acad Radiol 24:401–410

    Article  PubMed  Google Scholar 

  30. Maiga AW, Deppen SA, Massion PP et al (2018) Communication about the probability of cancer in indeterminate pulmonary nodules. JAMA Surg 153:353–357

    Article  PubMed  Google Scholar 

  31. Dilger SK, Uthoff J, Judisch A et al (2015) Improved pulmonary nodule classification utilizing quantitative lung parenchyma features. J Med Imaging (Bellingham) 2:041004

    Article  Google Scholar 

  32. Dhara AK, Mukhopadhyay S, Dutta A, Garg M, Khandelwal N (2016) A combination of shape and texture features for classification of pulmonary nodules in lung CT images. J Digit Imaging 29:466–475

    Article  PubMed  PubMed Central  Google Scholar 

  33. Ferreira JR Jr, Oliveira MC, de Azevedo-Marques PM (2017) Characterization of pulmonary nodules based on features of margin sharpness and texture. J Digit Imaging 31:451–463

  34. Nibali A, He Z, Wollersheim D (2017) Pulmonary nodule classification with deep residual networks. Int J Comput Assist Radiol Surg 12:1799–1808

    Article  PubMed  Google Scholar 

  35. Sun W, Zheng B, Qian W (2017) Automatic feature learning using multichannel ROI based on deep structured algorithms for computerized lung cancer diagnosis. Comput Biol Med 89:530–539

    Article  PubMed  Google Scholar 

  36. Way TW, Sahiner B, Chan HP et al (2009) Computer-aided diagnosis of pulmonary nodules on CT scans: improvement of classification performance with nodule surface features. Med Phys 36:3086–3098

    Article  PubMed  PubMed Central  Google Scholar 

  37. Zhu Y, Tan Y, Hua Y, Wang M, Zhang G, Zhang J (2010) Feature selection and performance evaluation of support vector machine (SVM)-based classifier for differentiating benign and malignant pulmonary nodules by computed tomography. J Digit Imaging 23:51–65

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We thank Kimberly Sprenger, Debra O’Connel-Moore, Mark Escher, Patrick Thalken, and Kimberly Schroeder for technical assistance.

Funding

This work was supported in part by Grant IRG-77-004-34 from the American Cancer Society, administered through the Holden Comprehensive Cancer Center at the University of Iowa. The COPDGene Study was supported by NHLBI U01 HL089897 and U01 HL089856. The COPDGene study (NCT00608764) is also supported by the COPD Foundation through contributions made to an Industry Advisory Committee comprised of AstraZeneca, Boehringer-Ingelheim, GlaxoSmithKline, Novartis, Pfizer, Siemens, and Sunovion. The INHALE study was supported by Award Number R01CA141769 and P30CA022453 from the National Cancer Institute, Health and Human Services Award HHSN26120130011I, and the Herrick Foundation.

Author information

Authors and Affiliations

  1. Department of Radiology, University of Iowa, 200 Hawkins Drive cc704 GH, Iowa City, IA, 52242, USA

    Johanna Uthoff, Nicholas Koehn, Jared Larson, Samantha K. N. Dilger, Emily Hammond, Brian Mullan & Jessica C. Sieren

  2. Department of Biomedical Engineering, University of Iowa, 5601 Seamans Center, Iowa City, IA, 52242, USA

    Johanna Uthoff, Samantha K. N. Dilger, Emily Hammond & Jessica C. Sieren

  3. Karmanos Cancer Institute, Wayne State University, 4100 John R St, Detroit, MI, 48201, USA

    Ann Schwartz

  4. Department of Internal Medicine, University of Iowa, 200 Hawkins Drive, Iowa City, IA, 52242, USA

    Rolando Sanchez & Richard M. Hoffman

Authors
  1. Johanna Uthoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas Koehn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jared Larson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Samantha K. N. Dilger
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emily Hammond
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ann Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Brian Mullan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rolando Sanchez
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Richard M. Hoffman
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jessica C. Sieren
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

for the COPDGene Investigators

Corresponding author

Correspondence to Jessica C. Sieren.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Dr. Jessica C. Sieren.

Conflict of interest

The authors declare that they have no conflict of interest.

Statistics and biometry

The first and last authors, as biomedical engineers, have experience with biostatistics methods. No complex statistical methods were necessary for this paper.

Informed consent

The University of Iowa Institutional Review Board has approved this study (IRB 201603824). Informed consent was obtained from the research cohort participants through the parent studies, COPDgene and INHALE (including the approval of collected data for expanded research questions beyond the parent study purpose). For the retrospective clinical cohort, written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

Forty subjects from study subjects or cohorts have been previously reported by our lab in a machine learning approach development [19, 20, 31].

Methodology

• Retrospective

• observational

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(DOCX 225 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Uthoff, J., Koehn, N., Larson, J. et al. Post-imaging pulmonary nodule mathematical prediction models: are they clinically relevant?. Eur Radiol 29, 5367–5377 (2019). https://doi.org/10.1007/s00330-019-06168-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-019-06168-x

Keywords

Search

Navigation