Next Article in Journal
Gastrointestinal Dysfunction in Parkinson’s Disease: Current and Potential Therapeutics
Previous Article in Journal
A Non-Linear Relationship between Preoperative Total Bilirubin Level and Postoperative Delirium Incidence after Liver Transplantation
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JPM
Volume 12
Issue 2
10.3390/jpm12020143
jpm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Locoregional Recurrence Prediction Using a Deep Neural Network of Radiological and Radiotherapy Images

by
Kyumin Han
1,2,†,
Joonyoung Francis Joung
3,
Minhi Han
3,
Wonmo Sung
1,4,* and
Young-nam Kang
2,5,*,†
1
Department of Biomedicine & Health Sciences, College of Medicine, The Catholic University of Korea, Seoul 06591, Korea
2
Advanced Institute for Radiation Fusion Medical Technology, College of Medicine, The Catholic University of Korea, Seoul 06591, Korea
3
Department of Chemistry and Research, Institute for Natural Science, Korea University, Seoul 02841, Korea
4
Department of Biomedical Engineering, College of Medicine, The Catholic University of Korea, Seoul 06591, Korea
5
Department of Radiation Oncology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul 06591, Korea
*
Authors to whom correspondence should be addressed.
†
These authors contributed equally.
J. Pers. Med. 2022, 12(2), 143; https://doi.org/10.3390/jpm12020143
Submission received: 12 October 2021 / Revised: 8 January 2022 / Accepted: 10 January 2022 / Published: 21 January 2022
(This article belongs to the Section Omics/Informatics)
Browse Figures
Versions Notes

Abstract

:
Radiation therapy (RT) is an important and potentially curative modality for head and neck squamous cell carcinoma (HNSCC). Locoregional recurrence (LR) of HNSCC after RT is ranging from 15% to 50% depending on the primary site and stage. In addition, the 5-year survival rate of patients with LR is low. To classify high-risk patients who might develop LR, a deep learning model for predicting LR needs to be established. In this work, 157 patients with HNSCC who underwent RT were analyzed. Based on the National Cancer Institute’s multi-institutional TCIA data set containing FDG-PET/CT/dose, a 3D deep learning model was proposed to predict LR without time-consuming segmentation or feature extraction. Our model achieved an averaged area under the curve (AUC) of 0.856. Adding clinical factors into the model improved the AUC to an average of 0.892 with the highest AUC of up to 0.974. The 3D deep learning model could perform individualized risk quantification of LR in patients with HNSCC without time-consuming tumor segmentation.

1. Introduction

More than 650,000 cases of head and neck cancer including head and neck squamous cell carcinoma (HNSCC) have been reported worldwide annually, making it the seventh most common cancer [1,2]. To treat HNSCC, radiation therapy is frequently considered [3]. The locoregional recurrence (LR) varies depending on the primary site and stage and ranging from 15% to 50% after treatment of HNSCC [4,5,6,7,8]. In addition, the outcome of patients with LR after treatment is poor, with a 5-year survival rate of 50% [4,5,6]. To find high-risk patients who might suffer from LR in advance, it is necessary to develop a model capable of predicting the LR.
To predict the LR, radiomics has been widely used and proved to be powerful for predicting the prognosis of tumor after it is treated [8,9,10,11,12,13,14,15,16,17]. In addition, the prediction performance can be further improved using combined radiomics and dosiomics [16]. Although tumor segmentation and feature extraction are essential parts of -omics prediction, they are not integrated into a single model. Consequently, such -omics methods are hardly reproducible in other institutions [17,18,19,20]. Therefore, it is necessary to deal with radiological and radiotherapy images themselves, instead of radiomics and dosiomics.
A deep neural network (DNN) has been attracting attention as an innovative method for image analysis, natural language processing, and various research methods [12,21,22,23,24,25], because it can learn the rules governing the underlying phenomena directly from the data. Therefore, a solution for such problems can be solved by using a deep neural network. Combining DNNs with radiomics shows great potential to predict cancer patient prognosis following treatment [15,26]. One advantage of DNNs is that a DNN does not need a process of segmentation or feature extraction because DNN can capture features from images by training. Moreover, a DNN has the ability to extract features that cannot be described by traditional analytic algorithms.
To predict the prognosis of HNSCC, several methods based on radiomics have been reported [14,15]. Such methods based on radiomics have demonstrated that LR, distant metastasis, and overall survival of HNSCC can be predicted. In these studies, predicting models were evaluated using area under the curve (AUC) of receiver operating characteristic (ROC) curves. Diamant et al. have reported that they could achieve an AUC of 0.92 for predicting distant metastasis [15]. However, the AUC for predicting LR has been to be 0.65 by Diamant et al. and 0.69 by Vallières et al. [14,15], indicating that predicting distant metastasis and overall survival is more reliable than predicting LR. As AUC for predicting LR was found to be relatively low, the performance of predicting LR remains an issue.
In this work, we proposed a convolutional neural network (CNN) to predict the LR of patients with HNSCC following radiation therapy [1,3,12,27,28,29,30]. CNN-based deep learning models were designed to be capable of predicting LR using CT, FDG-PET, dose distribution, or clinical factors. In our model, medical images and does distributions were not segmented. Their features were not extracted to avoid the issue related to reproducibility [17,18,19,20]. In addition, five different models with different input configurations were compared. We found that the deep learning model receiving three images and additional clinical factors as input could predict the LR most effectively. This will be discussed below in great detail.

2. Materials and Methods

Data set: We used the data set containing FDG-PET/CT and radiotherapy planning CT imaging data of 298 patients with HNSCC in The Cancer Imaging Archive (TCIA) [14]. After excluding incomplete data set, data of 157 patients were used for this study as summarized in Table S1. For cross-validation, the data set was randomly divided into a training set (134 patients) and a test set (23 patients) for 5 times as summarized in Table S1. p-values were calculated by Fisher’s exact test to compare training and validation data sets.
Image preprocessing: CT and PET images and radiation dose distribution were cropped and interpolated into a total size of 300 mm × 300 mm × 99 mm with a voxel size of 3 mm × 3 mm × 3 mm. Homemade MATLAB® (The MathWorks Inc., Natick, MA, USA) codes were used to crop and interpolate them.
Deep learning model: We developed a three-dimensional deep learning architecture as shown in Figure 1. We proposed three types of 3D DL architectures depending on the number of inputs (two, three, and four major inputs). Two inputs architectures were CP (CT+PET), CD (CT+Dose), and PD (PET+Dose) models (Figure 1a and Table S2). We also considered architectures (CPD model: CT+PET+Dose) with three inputs (Figure 1b). Finally, the CPD-C (CT+PET+Dose+clinical factors) architectures in Figure 1c included baseline patient characteristics such as sex, age, tumor stages, and primary disease site.
As CT and PET images with dose distribution were three dimensional (3D) images, 3D convolutional neural network (3D CNN) was used. For each model, conv3D and max pooling layers had window sizes of 3 × 3 × 2 and 2 × 2 × 2, respectively. After passing the first con3D and max pooling layers, hidden layers were summed to integrate information. To perform deep extraction of various features from inputs, channels of conv3D were set to be 16, 32, 64, 128, and 256. After setting all con3D and max pooling layers, vectors were flattened and passed through the one multi-layer perceptron to predict LR. In the CPD-C model, additional inputs of clinical factors were vectorized using the one-hot encoding method. Those inputs were integrated as shown in Figure 1c.
A rectified linear unit (ReLU) was used for activation functions of all layers except the last layer to predict LR. The activation function of the last layer was set to have a sigmoid function. Total parameters of models in Figure 1a–c were about 911, 911, and 915 thousand, respectively.
To train 3D DL models, images of a total of 157 patients were randomly divided into a training set (134 patients) and a test set (23 patients). As the number of images was relatively small, images were flipped in left–right, anterior–posterior, and inferior–superior directions. All combinations of flipping directions were applied to produce 8 times larger training data sets. Such data augmentation is a useful strategy to deal with small data set [31]. Three-dimensional DL models were optimized until the minimum validation loss was achieved as shown in Figure S1. Five randomly divided sets were used to train and test 3D DL models. No significant difference (p-value > 0.10) was found between the training and test sets for each data set by calculating Fisher’s exact test. Our 3D DL models were built using the Keras and trained using a computer with an intel CPU (i7-6700) and 16 GB of RAM.

3. Results

3.1. Features Captured by 3D DL Models

In the case of DL model using CNN, each channel in 3D convolutional (conv3D) layers could capture features in images. Figure 2 depicts several selected activation maps captured by conv3D layers in the CPD-C model. A close inspection of feature map revealed the area around the tumor (highlighted). In the first row of the first CNN layer, activation maps resulting from CT images as input could capture overall shape and boundary. However, the area around the tumor was highlighted in activation maps by the layer that passed PET images. Activation maps produced by dose distributions captured a relatively complex pattern in the dose distribution. Therefore, our 3D DL models were found to have the ability to capture tumor related features to predict LR. As shown in Figures S2–S6, activation maps tuned into a more abstract representation to relate LR.

3.2. LR Prediction Using 3D DL Models

The most accurate 3D DL model was found to be the CPD-C model with an average accuracy of 88.7% ± 4%. In addition, our 3D DL model required only 1.2 s to predict the LR after training using a computer with an intel CPU (i7-6700) and 16 GB of RAM. Three-dimensional DL models were further evaluated using area under the curve (AUC) of receiver operating characteristic (ROC) curves. The ROC curve is a graph of the false positive rate and the true positive rate obtained by changing the threshold value, which is the criterion for distinguishing positive and negative. Since our DL model was a binary classifier that distinguished whether or not LR occurred, the AUC of the ROC curve was used to compare five different 3D DL models. Results are shown in Figure 3. The CPD-C model was found to have the largest average AUC of 0.892 ± 0.07. AUCs were gradually increased in the order of CD (0.72 ± 0.04), CP (0.77 ± 0.07), PD (0.83 ± 0.07), and CPD (0.86 ± 0.07) models.

4. Discussion

Several studies have predicted the LR of HNSCC patients using various radiomics methods as summarized in Table 1. After feature extraction from CT and PET images, a method of finding the correlation with LR using a multivariate Cox proportional hazard regression model has been reported [16]. In this method, concordance index (CI) values were found to be 0.60, 0.66, and 0.56 for CP, CPD, and CPD-C models, respectively. Meanwhile, Wang et al. have reported a machine learning model such as a support vector machine using extracted features to predict LR [8]. In their study, AUCs of CP and CP-C were 0.76 and 0.77, respectively. In the group reporting the HNSCC data set to TCIA, AUC values of 0.64 and 0.69 were reported when LR was predicted with a random forest model using CP and CP-C [14]. Methods attempted to predict LR so far have been based on feature extraction. Recently, Diamant et al. have reported a CNN model for predicting the LR from central tumor slice, with an AUC of 0.65 [15].
Radiomic features such as first-order statistics and shape features are extracted after segmentation [16]. However, there is a risk of being missed if there is an unexpected effect between divided areas. In addition, unexpected image distortion may in discretizing CT and PET image with continuous values for feature extraction. Furthermore, features may have relatively shallow information because feature values are calculated according to a predetermined algorithm. However, features extracted by 3D DL model can be considered as more complex features than traditional features. Moreover, features are mixed to produce more abstract and informational features by passing through conv3D layers. After passing through the last con3D layer, features are transformed into the most abstract representation as shown in Figure S6.
In this study, the use of a 3D DL model was able to achieve a higher AUC than the conventional method. In the case of CP model, the averaged AUC value was 0.77 ± 0.07, which was higher than that of 0.76 of Wang et al. and 0.64 of M. Vallières et al. [8,14]. Such a high AUC value was achieved by skipping segmentation process and replacing feature extraction with deep learning. Since segmentation might suffer from inter- and intra-observer variation, it is a huge advance to avoid potentially erroneous procedures and human intervention [17,18,19,20]. Furthermore, an additional benefit of skipping the segmentation process is that the time needed for prediction can be dramatically reduced to about a second. Considering the time for segmentation, our model takes a negligible amount of time to make a prediction.
The AUC of the CPD model was larger than that of the CP model. This means that dose distribution with information about the treatment plan plays an important role in predicting recurrence after radiation therapy. As reported by Song and coworkers, dose distribution plays a very important role in predicting LR [16]. As shown by Song and coworkers, not only radiomics based on CT and PET images, but also dosiomics using the same strategy for dose distribution in predicting LR are important.
The average AUC value of the CD model shown in Table 1 was found to be lower than those of CP and PD models. As PET images clearly indicate the metabolic characterization of tumoral microenvironments, PET images may play a substantial role in predicting LR [12,32,33]. As averaged AUC values were increased in the order of CP, CPD, and CPD-C, LR prediction was improved when the more information was given. In the case of the CPD-C model to which the clinical factor was added, the averaged AUC value was 0.892 ± 0.07, which was the highest among all other models. In addition, the highest AUC of 0.974 was achieved for the CPD-C model. When compared to the concordance index (CI) value of CPD-C model of Song and coworkers (CI = 0.56) [16], the averaged AUC value of our CPD-C model is much higher. This is because our 3D DL model can effectively handle the clinical factors by representing them into more appropriate features in dense_1 to dense_6 layers and learning the correlations between the clinical factors and images in dense_7 and dense_9 layers as summarized in Table S4. Considering that the AUC of the CPD model was 0.856 ± 0.07, the improvement of AUC by clinical factors indicates that important information for prediction is not present in medical images or dose distribution. Additionally, the averaged AUC values of CPD-C model for each primary site and T stage are investigated as summarized in Table S5. The averaged AUC values of oropharynx and T2 stage were found to be highest among primary sites and T stages, respectively. This is because our 3D DL model was able to learn effectively based on the largest number of datapoints of oropharynx (n = 109) and T2 stage (n = 57).
DL models are often criticized in that it is hard to understand reason for the prediction the DL model made. This critique is called the Clever Hans effect, in which the DL model generates a correct answer for the wrong reason [34]. However, in the case of CNNs, the DL model shows an important part in making a prediction. Figure 4 shows the importance at the max_pooling3d_6 layer of the CPD-C model using Grad-CAM [35]. Importance values can be assigned by using the gradient information flowing into the last convolutional layer of the CNN in Grad-CAM [35].
Figure 4 depicts the contour and heatmap of an LR positive patient. The CPD-C model successfully predicted that the patient would be LR positive from images shown in Figure 4a–c. The red region in Figure 4f indicates where the important region captured by CPD-C model to predict LR. As shown in Figure 4g,h, the CPD-C model could recognize the area including the tumor area. In addition, it could be seen that a larger area including the tumor area was marked as important to predict LR. This indicates that not only the gross tumor volume (GTV), but also surrounding tissues are important for making a decision.
In the case of the CPD model, the CPD model accurately predicted that the patient would be LR negative from images shown in Figure 5a–c. Similarly, the CPD model captured a larger area including the tumor area. Figures S7 and S8 depict the heat map of patients who are LR positive and negative, respectively. This indicates that our 3D DL model can make a decision by referring to complex features on images. In conventional radiomics, feature extractions were performed in the GTV area. It might miss information for surrounding tissues of tumor. However, 3D DL models showed that our model could overcome such issues by considering images themselves without segmentation or extracting complex features directly from images.
In our model, only six clinical factors (sex, age, T stage, N stage, M stage, and primary site) were considered to predict LR. In fact, human papilloma virus (HPV) is known to be related to HNSCC [12,36,37]. However, the infection of HPV was not handled in our model because information about HPV was lacking in the data set. The status of HPV infection might be important to improve our 3D DL model. In addition, there could be other unknown clinical factors, such as effects of chemotherapy, tobacco consumption, and alcohol abuse, related to LR of HNSCC. Deep learning method is useful to identify which clinical factors are important to improve model performance. Systematic research on finding critical clinical factors can open the opportunity to find high risk patients.

5. Conclusions

This study demonstrated the impact of deep learning on LR prediction through pretreatment images (CT/PET scans) and treatment schemes (dose distributions). The performance of LR prediction was increased when baseline patient characteristics were considered. Our model allowed prediction consistency and established a cost-effective system to meet large clinical demands with less manual intervention. The use of a non-invasive model with pretreatment information could predict patient prognosis, which can have a potential clinical implication on personalized therapy.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/jpm12020143/s1, Figure S1: The training procedure of CPD-C model with set 1, Figures S2–S6: activation maps passing through the layers of CPD-C model in Table S3, and Figures S7 and S8: Heat map of CPD-C model for the images of patient, Table S1: The characteristics of 157 patients in head and neck squamous cell carcinoma data set., Tables S2–S4: The architecture of deep learning models, Table S5: Averaged AUC of CPD-C model for each primary site, T and N stage.

Author Contributions

K.H. and J.F.J. contributed equally. Conceptualization, K.H., J.F.J. and Y.-n.K.; methodology, K.H. and M.H.; analysis, K.H.; supervision, J.F.J. and Y.-n.K.; writing, K.H., W.S., J.F.J. and Y.-n.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by National Research Foundation of Korea (NRF) grant number 2021R1C1C1005930 and the APC was funded by Catholic University of Korea.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data set containing FDG-PET/CT and radiotherapy planning CT imaging data of 298 patients with HNSCC can be found at The Cancer Imaging Archive (TCIA) (https://doi.org/10.7937/K9/TCIA.2017.8oje5q00, accessed on 11 October 2021).

Acknowledgments

This study was supported by Advanced Institute for Radiation Fusion Medical Technology (AIRFMT) at Catholic University of Korea. This work was also supported by a grant (2021R1C1C1005930) from the National Research Foundation of Korea (NRF) funded by the Korea government (MSIT).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Bray, F.; Ferlay, J.; Soerjomataram, I.; Siegel, R.L.; Torre, L.A.; Jemal, A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 2018, 68, 394–424. [Google Scholar] [CrossRef] [Green Version]
  2. Johnson, D.E.; Burtness, B.; Leemans, C.R.; Lui, V.W.Y.; Bauman, J.E.; Grandis, J.R. Head and neck squamous cell carcinoma. Nat. Rev. Dis. Prim. 2020, 6, 92. [Google Scholar] [CrossRef] [PubMed]
  3. Atun, R.; Jaffray, D.A.; Barton, M.B.; Bray, F.; Baumann, M.; Vikram, B.; Hanna, T.P.; Knaul, F.M.; Lievens, Y.; Lui, T.Y.M.; et al. Expanding global access to radiotherapy. Lancet Oncol. 2015, 16, 1153–1186. [Google Scholar] [CrossRef]
  4. Ferlito, A.; Shaha, A.R.; Silver, C.E.; Rinaldo, A.; Mondin, V. Incidence and sites of distant metastases from head and neck cancer. ORL J. Otorhinolaryngol. Relat. Spec. 2001, 63, 202–207. [Google Scholar] [CrossRef]
  5. Yeh, S.A. Radiotherapy for head and neck cancer. Semin. Plast. Surg. 2010, 24, 127–136. [Google Scholar] [CrossRef] [Green Version]
  6. Baxi, S.S.; Pinheiro, L.C.; Patil, S.M.; Pfister, D.G.; Oeffinger, K.C.; Elkin, E.B. Causes of death in long-term survivors of head and neck cancer. Cancer 2014, 120, 1507–1513. [Google Scholar] [CrossRef] [Green Version]
  7. Keek, S.; Sanduleanu, S.; Wesseling, F.; de Roest, R.; van den Brekel, M.; van der Heijden, M.; Vens, C.; Giuseppina, C.; Licitra, L.; Scheckenbach, K.; et al. Computed tomography-derived radiomic signature of head and neck squamous cell carcinoma (peri)tumoral tissue for the prediction of locoregional recurrence and distant metastasis after concurrent chemo-radiotherapy. PLoS ONE 2020, 15, e0232639. [Google Scholar] [CrossRef]
  8. Wang, K.; Zhou, Z.G.; Wang, R.F.; Chen, L.Y.; Zhang, Q.W.; Sher, D.; Wang, J. A multi-objective radiomics model for the prediction of locoregional recurrence in head and neck squamous cell cancer. Med. Phys. 2020, 47, 5392–5400. [Google Scholar] [CrossRef] [PubMed]
  9. Lambin, P.; Rios-Velazquez, E.; Leijenaar, R.; Carvalho, S.; van Stiphout, R.G.; Granton, P.; Zegers, C.M.; Gillies, R.; Boellard, R.; Dekker, A.; et al. Radiomics: Extracting more information from medical images using advanced feature analysis. Eur. J. Cancer 2012, 48, 441–446. [Google Scholar] [CrossRef] [Green Version]
  10. Yip, S.S.; Aerts, H.J. Applications and limitations of radiomics. Phys. Med. Biol. 2016, 61, R150–R166. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. Mayerhoefer, M.E.; Materka, A.; Langs, G.; Haggstrom, I.; Szczypinski, P.; Gibbs, P.; Cook, G. Introduction to Radiomics. J. Nucl. Med. 2020, 61, 488–495. [Google Scholar] [CrossRef] [PubMed]
  12. Men, K.; Geng, H.; Zhong, H.; Fan, Y.; Lin, A.; Xiao, Y. A Deep Learning Model for Predicting Xerostomia Due to Radiation Therapy for Head and Neck Squamous Cell Carcinoma in the RTOG 0522 Clinical Trial. Int. J. Radiat. Oncol. Biol. Phys. 2019, 105, 440–447. [Google Scholar] [CrossRef]
  13. Lao, J.; Chen, Y.; Li, Z.C.; Li, Q.; Zhang, J.; Liu, J.; Zhai, G. A Deep Learning-Based Radiomics Model for Prediction of Survival in Glioblastoma Multiforme. Sci. Rep. 2017, 7, 10353. [Google Scholar] [CrossRef]
  14. Vallieres, M.; Kay-Rivest, E.; Perrin, L.J.; Liem, X.; Furstoss, C.; Aerts, H.; Khaouam, N.; Nguyen-Tan, P.F.; Wang, C.S.; Sultanem, K.; et al. Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Sci. Rep. 2017, 7, 10117. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Diamant, A.; Chatterjee, A.; Vallieres, M.; Shenouda, G.; Seuntjens, J. Deep learning in head & neck cancer outcome prediction. Sci. Rep. 2019, 9, 2764. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Wu, A.; Li, Y.; Qi, M.; Lu, X.; Jia, Q.; Guo, F.; Dai, Z.; Liu, Y.; Chen, C.; Zhou, L.; et al. Dosiomics improves prediction of locoregional recurrence for intensity modulated radiotherapy treated head and neck cancer cases. Oral. Oncol. 2020, 104, 104625. [Google Scholar] [CrossRef]
  17. Yang, F.; Simpson, G.; Young, L.; Ford, J.; Dogan, N.; Wang, L. Impact of contouring variability on oncological PET radiomics features in the lung. Sci. Rep. 2020, 10, 369. [Google Scholar] [CrossRef]
  18. Qiu, Q.; Duan, J.; Duan, Z.; Meng, X.; Ma, C.; Zhu, J.; Lu, J.; Liu, T.; Yin, Y. Reproducibility and non-redundancy of radiomic features extracted from arterial phase CT scans in hepatocellular carcinoma patients: Impact of tumor segmentation variability. Quant. Imaging Med. Surg. 2019, 9, 453–464. [Google Scholar] [CrossRef]
  19. Pfaehler, E.; Burggraaff, C.; Kramer, G.; Zijlstra, J.; Hoekstra, O.S.; Jalving, M.; Noordzij, W.; Brouwers, A.H.; Stevenson, M.G.; de Jong, J.; et al. PET segmentation of bulky tumors: Strategies and workflows to improve inter-observer variability. PLoS ONE 2020, 15, e0230901. [Google Scholar] [CrossRef]
  20. Lindberg, K.; Kouti, A.; Ziegelitz, D.; Hallen, T.; Skoglund, T.; Farahmand, D. Three-Dimensional Volumetric Segmentation of Pituitary Tumors: Assessment of Inter-rater Agreement and Comparison with Conventional Geometric Equations. J. Neurol. Surg. B Skull Base 2018, 79, 475–481. [Google Scholar] [CrossRef]
  21. Litjens, G.; Kooi, T.; Bejnordi, B.E.; Setio, A.A.A.; Ciompi, F.; Ghafoorian, M.; van der Laak, J.; van Ginneken, B.; Sanchez, C.I. A survey on deep learning in medical image analysis. Med. Image Anal. 2017, 42, 60–88. [Google Scholar] [CrossRef] [Green Version]
  22. Parekh, V.S.; Jacobs, M.A. Deep learning and radiomics in precision medicine. Expert Rev. Precis Med. Drug Dev. 2019, 4, 59–72. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  23. Avanzo, M.; Wei, L.; Stancanello, J.; Vallieres, M.; Rao, A.; Morin, O.; Mattonen, S.A.; El Naqa, I. Machine and deep learning methods for radiomics. Med. Phys. 2020, 47, e185–e202. [Google Scholar] [CrossRef]
  24. Xu, Y.; Vaziri-Pashkam, M. Limits to visual representational correspondence between convolutional neural networks and the human brain. Nat. Commun. 2021, 12, 2065. [Google Scholar] [CrossRef]
  25. Willett, F.R.; Avansino, D.T.; Hochberg, L.R.; Henderson, J.M.; Shenoy, K.V. High-performance brain-to-text communication via handwriting. Nature 2021, 593, 249–254. [Google Scholar] [CrossRef] [PubMed]
  26. Hosny, A.; Parmar, C.; Coroller, T.P.; Grossmann, P.; Zeleznik, R.; Kumar, A.; Bussink, J.; Gillies, R.J.; Mak, R.H.; Aerts, H. Deep learning for lung cancer prognostication: A retrospective multi-cohort radiomics study. PLoS Med. 2018, 15, e1002711. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  27. Brennan, S.; Corry, J.; Kleid, S.; Porceddu, S.; Yuen, K.; Rischin, D.; Peters, L.J. Prospective trial to evaluate staged neck dissection or elective neck radiotherapy in patients with CT-staged T1-2 N0 squamous cell carcinoma of the oral tongue. Head Neck 2010, 32, 191–198. [Google Scholar] [CrossRef] [PubMed]
  28. Caudell, J.J.; Ward, M.C.; Riaz, N.; Zakem, S.J.; Awan, M.J.; Dunlap, N.E.; Isrow, D.; Hassanzadeh, C.; Vargo, J.A.; Heron, D.E.; et al. Volume, Dose, and Fractionation Considerations for IMRT-based Reirradiation in Head and Neck Cancer: A Multi-institution Analysis. Int. J. Radiat. Oncol. Biol. Phys. 2018, 100, 606–617. [Google Scholar] [CrossRef]
  29. Cramer, J.D.; Burtness, B.; Le, Q.T.; Ferris, R.L. The changing therapeutic landscape of head and neck cancer. Nat. Rev. Clin. Oncol. 2019, 16, 669–683. [Google Scholar] [CrossRef] [PubMed]
  30. Chow, L.Q.M. Head and Neck Cancer. N. Engl. J. Med. 2020, 382, 60–72. [Google Scholar] [CrossRef] [PubMed]
  31. Hao, R.; Namdar, K.; Liu, L.; Haider, M.A.; Khalvati, F. A Comprehensive Study of Data Augmentation Strategies for Prostate Cancer Detection in Diffusion-Weighted MRI Using Convolutional Neural Networks. J. Digit. Imaging 2021, 34, 862–876. [Google Scholar] [CrossRef] [PubMed]
  32. Schoder, H.; Yeung, H.W.; Gonen, M.; Kraus, D.; Larson, S.M. Head and neck cancer: Clinical usefulness and accuracy of PET/CT image fusion. Radiology 2004, 231, 65–72. [Google Scholar] [CrossRef] [PubMed]
  33. Al-Ibraheem, A.; Buck, A.; Krause, B.J.; Scheidhauer, K.; Schwaiger, M. Clinical Applications of FDG PET and PET/CT in Head and Neck Cancer. J. Oncol. 2009, 2009, 208725. [Google Scholar] [CrossRef] [Green Version]
  34. Lapuschkin, S.; Waldchen, S.; Binder, A.; Montavon, G.; Samek, W.; Muller, K.R. Unmasking Clever Hans predictors and assessing what machines really learn. Nat. Commun. 2019, 10, 1096. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  35. Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int. J. Comput. Vis. 2019, 128, 336–359. [Google Scholar] [CrossRef] [Green Version]
  36. Dayyani, F.; Etzel, C.J.; Liu, M.; Ho, C.H.; Lippman, S.M.; Tsao, A.S. Meta-analysis of the impact of human papillomavirus (HPV) on cancer risk and overall survival in head and neck squamous cell carcinomas (HNSCC). Head Neck Oncol. 2010, 2, 15. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  37. Gronhoj, C.; Jakobsen, K.K.; Jensen, D.H.; Rasmussen, J.; Andersen, E.; Friborg, J.; von Buchwald, C. Pattern of and survival following loco-regional and distant recurrence in patients with HPV+ and HPV-oropharyngeal squamous cell carcinoma: A population-based study. Oral. Oncol. 2018, 83, 127–133. [Google Scholar] [CrossRef] [PubMed]
Jpm 12 00143 g001 550
Figure 1. Deep learning model for predicting locoregional recurrence (LR): (a) DL model for CP (CT+PET) model. (b) CPD (CT+PET+Dose) model. (c) CPD-C (CT+PET+Dose+clinical factors) model. Architectures of DL models in detail can be found in Supporting Information.
Figure 1. Deep learning model for predicting locoregional recurrence (LR): (a) DL model for CP (CT+PET) model. (b) CPD (CT+PET+Dose) model. (c) CPD-C (CT+PET+Dose+clinical factors) model. Architectures of DL models in detail can be found in Supporting Information.
Jpm 12 00143 g001
Jpm 12 00143 g002 550
Figure 2. Visualized activation maps of the CPD-C (CT+PET+Dose+clinical factors) model. The first, second, and third rows of activation maps in the 1st CNN layer were extracted from conv3d_1, conv3d_2, and conv3d_3, respectively (See the architecture in Table S3). Activation maps in the 2nd CNN layer were extracted from conv3d_4.
Figure 2. Visualized activation maps of the CPD-C (CT+PET+Dose+clinical factors) model. The first, second, and third rows of activation maps in the 1st CNN layer were extracted from conv3d_1, conv3d_2, and conv3d_3, respectively (See the architecture in Table S3). Activation maps in the 2nd CNN layer were extracted from conv3d_4.
Jpm 12 00143 g002
Jpm 12 00143 g003 550
Figure 3. ROC curves of: (a) CP (CT+PET), (b) CD (CT+Dose), (c) PD (PET+Dose), (d) CPD (CT+PET+Dose), and (e) CPD-C (CT+PET+Dose+clinical factors) models.
Figure 3. ROC curves of: (a) CP (CT+PET), (b) CD (CT+Dose), (c) PD (PET+Dose), (d) CPD (CT+PET+Dose), and (e) CPD-C (CT+PET+Dose+clinical factors) models.
Jpm 12 00143 g003
Jpm 12 00143 g004 550
Figure 4. Heat map of LR positive patient produced by the CPD-C model: (a) Input images of CT. (b). Input images of PET. (c). Input images of dose distribution. (d). Contour images of CT. (e). Contour images of PET. (f). The heatmap. The red region in the heatmap indicates where the important region to make decision of LR. (g–i) are superimposed images of CT, PET, and dose distribution and heatmap, respectively.
Figure 4. Heat map of LR positive patient produced by the CPD-C model: (a) Input images of CT. (b). Input images of PET. (c). Input images of dose distribution. (d). Contour images of CT. (e). Contour images of PET. (f). The heatmap. The red region in the heatmap indicates where the important region to make decision of LR. (g–i) are superimposed images of CT, PET, and dose distribution and heatmap, respectively.
Jpm 12 00143 g004
Jpm 12 00143 g005 550
Figure 5. Heat map of an LR negative patient produced by the CPD model: (a). Input images of CT. (b). Input images of PET. (c). Input images of dose distribution. (d). Contour images of CT. (e). Contour images of PET. (f). The heatmap. The red region in the heatmap indicating the important region to make decision of LR. (g–i) are superimposed images of CT, PET, and dose distribution and heatmap, respectively.
Figure 5. Heat map of an LR negative patient produced by the CPD model: (a). Input images of CT. (b). Input images of PET. (c). Input images of dose distribution. (d). Contour images of CT. (e). Contour images of PET. (f). The heatmap. The red region in the heatmap indicating the important region to make decision of LR. (g–i) are superimposed images of CT, PET, and dose distribution and heatmap, respectively.
Jpm 12 00143 g005
Table
Table 1. Averaged AUC of test set compared to AUCs of other previous studies.
Table 1. Averaged AUC of test set compared to AUCs of other previous studies.
CPCP-CCDPDCPDCPD-C
Song et al. [16] 0.60 a---0.66 a0.56 a
Wang et al. [8]0.760.77----
Vallières et al. [14]0.640.69----
Diamant et al. [15]0.65-----
In this study0.77 ± 0.07-0.72 ± 0.040.83 ± 0.070.86 ± 0.070.89 ± 0.07
a Those values are concordance index (CI) values.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Han, K.; Joung, J.F.; Han, M.; Sung, W.; Kang, Y.-n. Locoregional Recurrence Prediction Using a Deep Neural Network of Radiological and Radiotherapy Images. J. Pers. Med. 2022, 12, 143. https://doi.org/10.3390/jpm12020143

AMA Style

Han K, Joung JF, Han M, Sung W, Kang Y-n. Locoregional Recurrence Prediction Using a Deep Neural Network of Radiological and Radiotherapy Images. Journal of Personalized Medicine. 2022; 12(2):143. https://doi.org/10.3390/jpm12020143

Chicago/Turabian Style

Han, Kyumin, Joonyoung Francis Joung, Minhi Han, Wonmo Sung, and Young-nam Kang. 2022. "Locoregional Recurrence Prediction Using a Deep Neural Network of Radiological and Radiotherapy Images" Journal of Personalized Medicine 12, no. 2: 143. https://doi.org/10.3390/jpm12020143

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop