Abstract
Objectives
The prediction of primary treatment failure (PTF) is necessary for patients with diffuse large B-cell lymphoma (DLBCL) since it serves as a prominent means for improving front-line outcomes. Using interim 18F-fluoro-2-deoxyglucose ([18F]FDG) positron emission tomography/computed tomography (PET/CT) imaging data, we aimed to construct multimodal deep learning (MDL) models to predict possible PTF in low-risk DLBCL.
Methods
Initially, 205 DLBCL patients undergoing interim [18F]FDG PET/CT scans and the front-line standard of care were included in the primary dataset for model development. Then, 44 other patients were included in the external dataset for generalization evaluation. Based on the powerful backbone of the Conv-LSTM network, we incorporated five different multimodal fusion strategies (pixel intermixing, separate channel, separate branch, quantitative weighting, and hybrid learning) to make full use of PET/CT features and built five corresponding MDL models. Moreover, we found the best model, that is, the hybrid learning model, and optimized it by integrating the contrastive training objective to further improve its prediction performance.
Results
The final model with contrastive objective optimization, named the contrastive hybrid learning model, performed best, with an accuracy of 91.22% and an area under the receiver operating characteristic curve (AUC) of 0.926, in the primary dataset. In the external dataset, its accuracy and AUC remained at 88.64% and 0.925, respectively, indicating its good generalization ability.
Conclusions
The proposed model achieved good performance, validated the predictive value of interim PET/CT, and holds promise for directing individualized clinical treatment.
Key Points
• The proposed multimodal models achieved accurate prediction of primary treatment failure in DLBCL patients.
• Using an appropriate feature-level fusion strategy can make the same class close to each other regardless of the modal heterogeneity of the data source domain and positively impact the prediction performance.
• Deep learning validated the predictive value of interim PET/CT in a way that exceeded human capabilities.
Similar content being viewed by others
Data availability
The data that support the findings of this study are available on reasonable request from the corresponding author.
Abbreviations
- [18F]FDG:
-
18F-fluoro-2-deoxyglucose
- AUC:
-
Area under the receiver operating characteristic curve
- DLBCL:
-
Diffuse large B-cell lymphoma
- ECOG:
-
Eastern Cooperative Oncology Group
- IQR:
-
Interquartile range
- LDH:
-
Lactate dehydrogenase
- LSTM:
-
Long short-term memory
- MDL:
-
Multimodal deep learning
- PET/CT:
-
Positron emission tomography/computed tomography
- PTF:
-
Primary treatment failure
- R-CHOP:
-
Rituximab, cyclophosphamide, doxorubicin, vincristine, and prednisone
- R-IPI:
-
Revised international prognostic index
- TMTV:
-
Total metabolic tumour volume
References
Sung H, Ferlay J, Siegel RL et al (2021) Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 71(3):209–249
Feugier P, Van Hoof A, Sebban C et al (2005) Long-term results of the R-CHOP study in the treatment of elderly patients with diffuse large B-cell lymphoma: a study by the Groupe d’Etude des Lymphomes de l’Adulte. J Clin Oncol 23:4117–4126
Crump M, Neelapu SS, Farooq U et al (2017) Outcomes in refractory diffuse large B-cell lymphoma: results from the international SCHOLAR-1 study. Blood 130(16):1800–1808
Gisselbrecht C, Neste EVD (2018) How I manage patients with relapsed/refractory diffuse large B cell lymphoma. Br J Haematol 182:633–643
Locke FL, Ghobadi A, Jacobson CA et al (2019) Long-term safety and activity of axicabtagene ciloleucel in refractory large B-cell lymphoma (ZUMA-1): a single-arm, multicentre, phase 1-2 trial. Lancet Oncol 20(1):31–42
Kalakonda N, Maerevoet M, Cavallo F et al (2020) Selinexor in patients with relapsed or refractory diffuse large B-cell lymphoma (SADAL): a single-arm, multinational, multicentre, open-label, phase 2 trial. Lancet Haematol 7(7):e511–e522
Hawkes EA, Barraclough A, Sehn LH (2022) Limited-stage diffuse large B-cell lymphoma. Blood 139(6):822–834
Lv X, Wang Q, Ge X, Xue C, Liu X (2021) Application of high-throughput gene sequencing in lymphoma. Exp Mol Pathol 119:104606
Sehn LH, Berry B, Chhanabhai M et al (2007) The revised International Prognostic Index (R-IPI) is a better predictor of outcome than the standard IPI for patients with diffuse large B-cell lymphoma treated with R-CHOP. Blood 190(6):1857–1861
Xu-Monette ZY, Wu L, Visco C et al (2012) Mutational profile and prognostic significance of TP53 in diffuse large B-cell lymphoma patients treated with R-CHOP: report from an international DLBCL rituximab-CHOP consortium program study. Blood 120(19):3986–3996
Juweid ME, Stroobants S, Hoekstra OS et al (2007) Use of positron emission tomography for response assessment of lymphoma: consensus of the Imaging Subcommittee of International Harmonization Project in Lymphoma. J Clin Oncol 25(5):571–578
Cheson BD, Pfistner B, Juweid ME et al (2007) Revised response criteria for malignant lymphoma. J Clin Oncol 25(5):579–586
Cheson BD (2011) Role of functional imaging in the management of lymphoma. J Clin Oncol 29(14):1844–1854
Jerusalem G, Beguin Y, Fassotte MF et al (2000) Persistent tumor [18F]FDG uptake after a few cycles of polychemotherapy is predictive of treatment failure in non-Hodgkin’s lymphoma. Haematologica 85(6):613–618
Spaepen K, Stroobants S, Dupont P et al (2002) Early restaging positron emission tomography with (18)F-fluorodeoxyglucose predicts outcome in patients with aggressive non-Hodgkin's lymphoma. Ann Oncol 13(9):1356–1363
Kirienko M, Biroli M, Gelardi F, Seregni E, Chiti A, Sollini M (2021) Deep learning in nuclear medicine—focus on CNN-based approaches for PET/CT and PET/MR: where do we stand? Clin Transl Imaging 9:37–55
Roth HR, Lu L, Seff A et al (2014) A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. Med Image Comput Comput Assist Interv 17:520–527
Wallis D, Soussan M, Lacroix M, Akl P, Duboucher C, Buvat I (2022) An [18F]FDG-PET/CT deep learning method for fully automated detection of pathological mediastinal lymph nodes in lung cancer patients. Eur J Nucl Med Mol Imaging 49:881–888
Bi L, Kim J, Kumar A, Wen L, Feng D, Fulham M (2017) Automatic detection and classification of regions of FDG uptake in whole-body PET-CT lymphoma studies. Comput Med Imaging Graph 60:3–10
Hu H, Shen L, Zhou T, Decazes P, Vera P, Ruan S (2020) Lymphoma segmentation in PET images based on multi-view and Conv3D fusion strategy. IEEE 17th International Symposium on Biomedical Imaging (ISBI):1197-1200
Revailler W, Cottereau AS, Rossi C et al (2022) Deep learning approach to automatize TMTV calculations regardless of segmentation methodology for major FDG-avid lymphomas. Diagnostics (Basel) 12(2):417
Sadik M, Lind E, Polymeri E, Enqvist O, Ulén J, Trägårdh E (2019) Automated quantification of reference levels in liver and mediastinal blood pool for the Deauville therapy response classification using FDG-PET/CT in Hodgkin and non-Hodgkin lymphomas. Clin Physiol Funct Imaging 39(1):78–84
Capobianco N, Meignan M, Cottereau AS et al (2021) Deep-learning [18F]FDG uptake classification enables total metabolic tumor volume estimation in diffuse large B-cell lymphoma. J Nucl Med 62(1):30–36
Seidler M, Forghani B, Reinhold C et al (2019) Dual-energy CT texture analysis with machine learning for the evaluation and characterization of cervical lymphadenopathy. Comput Struct Biotechnol J 17:1009–1015
Ganeshan B, Miles KA, Babikir S et al (2017) CT-based texture analysis potentially provides prognostic information complementary to interim FDG-PET for patients with Hodgkin’s and aggressive non-Hodgkin’s lymphomas. Eur Radiol 27:1012–1020
Santiago R, Jimenez JO, Forghani R et al (2021) CT-based radiomics model with machine learning for predicting primary treatment failure in diffuse large B-cell Lymphoma. Transl Oncol 14(10):101188
Zhou T, Ruan S, Canu S et al (2019) A review: Deep learning for medical image segmentation using multi-modality fusion. Array 3:100004
Li K, Zhang R, Cai W (2021) Deep learning convolutional neural network (DLCNN): unleashing the potential of 18 F-FDG PET/CT in lymphoma. Am J Nucl Med Mol Imaging 11(4):327–331
Jin C, Yu H, Ke J et al (2021) Predicting treatment response from longitudinal images using multi-task deep learning. Nat Commun 12:1851
Kumar A, Fulham M, Feng D, Kim J (2020) Co-learning feature fusion maps from PET-CT images of lung cancer. IEEE Trans Med Imaging 39(1):204–217
Donahue J, Hendricks LA, Rohrbach M et al (2017) Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans Pattern Anal Mach Intell 39(4):677–691
Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. Preprint arXiv:2002.05709
Zhong Z, Kim Y, Plichta K et al (2019) Simultaneous co-segmentation of tumors in PET-CT images using deep fully convolutional networks. Med Phys 46(2):619–633
Zhao X, Li L, Lu W, Tan S (2018) Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network. Phys Med Biol 64(1):015011
Humphries SM, Notary AM, Centeno JP et al (2019) Deep learning enables automatic classification of emphysema pattern at CT. Radiology 294(2):434–444
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Lu N, Wu Y, Feng L, Song J (2019) Deep learning for fall detection: three-dimensional CNN combined with LSTM on video kinematic data. IEEE J Biomed Health Inform 23(1):314–323
Abadi M, Barham P, Chen J, et al (2016) TensorFlow: a system for large-scale machine learning. Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation:265-283
Chetlur A, Woolley C, Vandermersch P, et al (2014) cuDNN: efficient primitives for deep learning. Preprint arXiv:1410.0759
Zhou B, Khosla A, Lapedriza A, Oliva A, Torrralba A (2016) Learning deep features for discriminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR):2921-2929
Du D, Feng H, Lv W et al (2020) Machine learning methods for optimal radiomics-based differentiation between recurrence and inflammation: application to nasopharyngeal carcinoma post-therapy PET/CT images. Mol Imaging Biol 22:730–738
Yuan C, Zhang M, Huang X et al (2021) Diffuse large B-cell lymphoma segmentation in PET-CT images via hybrid learning for feature fusion. Med Phys 48(7):3665–3678
Peng Y, Bi L, Guo Y, Feng D, Fulham M, Kim J (2019) Deep multi-modality collaborative learning for distant metastases predication in PET-CT soft-tissue sarcoma studies. Annu Int Conf IEEE Eng Med Biol Soc:3658-3688
Zhang W, Li R, Deng H et al (2015) Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. Neuroimage 108:214–224
Zhong Z, Kim Y, Zhou L, et al (2018) 3D fully convolutional networks for co-segmentation of tumors on PET-CT images. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI):228-231
Code availability
The code of our study is publicly accessible at https://github.com/cyuan-sjtu/MDL-model.
Funding
This study has received funding by the National Natural Science Foundation of China (81974276, 81830007, 81520108003, 81670176, and 82070204), Chang Jiang Scholars Program, Shanghai Municipal Education Commission Gaofeng Clinical Medicine Grant Support (20152206, and 20152208), Clinical Research Plan of Shanghai Hospital Development Center (SHDC2020CR1032B), Multicenter Clinical Research Project by Shanghai Jiao Tong University School of Medicine (DLY201601), Collaborative Innovation Center of Systems Biomedicine, and the Samuel Waxman Cancer Research Foundation.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Guarantor
The scientific guarantor of this publication is Dr. Dahong Qian.
Conflict of interest
The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.
Statistics and biometry
No complex statistical methods were necessary for this paper.
Informed consent
Written informed consent was not required for this study because of its retrospective nature.
Ethics approval
Institutional Review Board approval was obtained at Shanghai Ruijin Hospital.
Methodology
• Retrospective
• Diagnostic or prognostic study
• Performed at one institution
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(DOCX 94 kb)
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yuan, C., Shi, Q., Huang, X. et al. Multimodal deep learning model on interim [18F]FDG PET/CT for predicting primary treatment failure in diffuse large B-cell lymphoma. Eur Radiol 33, 77–88 (2023). https://doi.org/10.1007/s00330-022-09031-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00330-022-09031-8