2.1.

CBCT Data Acquisition and Preprocessing

All anonymized projection images from routine clinical radiotherapy CBCT scans from 1055 patients were collected over 24 nonconsecutive months. Because each patient can contribute scans from multiple treatment fractions, this resulted in a total of 6850 scans. The retrospective use of the data for this study was reviewed and approved by the institutional review board. All scans were acquired on TrueBeam linear accelerators (Varian Medical Systems, Palo Alto, California) using routine clinical protocols named for the anatomic class for which they are intended (see Appendix A). While some institutions manually alter protocol parameters for each patient, this is not our institutional practice.

Raw projections from 0 deg to 360 deg, at 45 deg intervals, were log normalized, then truncated to a given pixel value range (from 0.5 to 7.5) and mapped to 8-bit PNG images, using a MATLAB routine. The dynamic range was selected to include the majority of pixel values across the anatomic sites while excluding those outside the patient (i.e., $<0.5$). PNG images were used for labeling (Sec. 2.2) and for CNN training (Sec. 2.4). As others have done for medical image applications,^12,13 PNG images were rebinned to match the CNN architecture input size.¹⁴

2.2.

Image Labeling

CBCT scans were manually classified into five anatomic classes by a medical physicist with 20 years of experience in the practice of radiotherapy. A MATLAB interface was used to review all available projection images from 0 deg to 360 deg at 90 deg intervals, providing up to five projections per scan. Orthogonal views were used since they are more intuitive for a human observer and are thus standard in IGRT clinical workflow. Examples of projection images in each of the five classes with the midplane marked by a cyan line are shown in Fig. 1. The class selection was based on the mid-plane location relative to anatomical boundary lines. Upper and lower boundaries for each class are also listed in Fig. 1.

Fig. 1

Examples of projection images in each of the five classes with the mid-plane marked by a cyan line. Magenta lines indicate the boundaries between classes. Upper and lower anatomic landmarks used to determine the boundaries for each of the classes are indicated.

2.3.

Data Curation

There can be significant variability in the number of CBCT scans each patient receives based on the fractionation schedule and/or image frequency determined by their treatment protocol, e.g., a patient treated with hypofractionated stereotactic body radiotherapy may have only a few scans, whereas a prostate patient may have 10 times as many.⁶ This poses several challenges: (a) the similarity of the images from fraction to fraction may result in data that is highly redundant and (b) an imbalance in the proportion of data between classes may result. To mitigate this issue, a maximum number of scans per patient were included in the dataset used for model development: from 2 to 10 depending on the anatomic class. Ideally, 10 scans per patient would have been included for all classes but this was not possible for hypofractionated or single-fraction (i.e., stereotactic brain radiosurgery) treatments.

From each CBCT scan, projection images at every 45 deg (from 0 deg to 360 deg) were gathered. Projection data were grouped by view angle, resulting in eight different subsets. Each subset was divided as follows: 75% to train the model and 25% for testing and evaluation (equally partitioned), with no overlap of patients across training, testing, and evaluation datasets. Table 1 shows the resulting number of images per angular subset for each anatomic class as well as the total number of patients contributing to each class. Note that a range of images is given for head and neck classes because such scans were typically acquired with a head protocol, which utilizes a short scan (i.e., $\text{scan range}\approx 200\mathrm{deg}$) resulting in an angularly asymmetric dataset size.

Table 1

Total number of patients and images in the training, testing, and evaluation datasets for each anatomic class, given a projection angle. A range is reported for classes imaged with a shortened CBCT scan rotation (<360  deg).

2.4.

DL Model Development

Three CNNs were built using the Keras Deep Learning¹⁵ library with a TensorFlow¹⁶ backend engine (version 2.1.0, TensorFlow) to predict the anatomic class from a single projection image, given a fixed projection angle. These architectures, VGG-16,⁴ Xception,¹⁷ and Inception v3,¹⁸ were selected because they have been proven to be effective in the classification of natural^19–21 and medical images.^12,20,22,23 The classification layer of each network was adjusted to the number of classes in our classification task. All convolutional layers used rectified linear unit (ReLU) activation functions. CNNs were trained using the stochastic gradient descent optimizer, with a learning rate of 0.001, and transfer learning; that is, using the weights of a pretrained CNN (with the ImageNet dataset) as initial weights. Training was performed in two stages: (1) a warm-up stage (50 epochs), where all the weights of the networks were kept fixed, except for those of the last layer (classification layer), and (2) the training stage (100 epochs), where all weights were updated. Keras built-in data augmentation tools were used to prevent overfitting, allowing shifts (up to 10%) and zoom (up to 20%).

To study the dependence of the classification performance on dataset size, the training subset was downsampled to 5%, 10%, 15%, 20%, 30%, 50%, and 70% of its original size. For each training scheme, CNN training and evaluation was repeated 10 times to obtain the performance statistics. CNN training was performed with an NVIDIA GeForce RTX 2070 GPU and required about 1 h per repetition.

2.5.

Quantification of Classification Performance

The classification performance of the DL model was quantified in terms of precision (i.e., positive predictive value) and recall (i.e., sensitivity). A precision score of 1.0 means every item labeled as belonging to a class $C$ does indeed belong to class $C$, whereas a recall of 1.0 means every item from class $C$ was labeled as belonging to class $C$.

2.6.

Visualization of Features Identified by CNNs

Gradient-weighted class activation maps (grad-CAM)²⁴ were used to identify key features involved in the CNN classification process. A number of methods have been developed for visualizing pixel attribution (saliency maps), i.e., the regions of an image that are relevant for a CNN classification.^10,11,25 Grad-CAM was used here as it has been previously shown to be useful in the interpretation of medical image classification models.^10,11,25 Grad-CAM of images in the evaluation dataset were generated using the DL model with the highest precision and recall values. In brief, Grad-CAM illustrates the relative spatial activation of the final CNN layer (before classification) with respect to the network output. Grad-CAM were computed as ReLU activations of a weighted sum of feature maps obtained from the final CNN layer, where sum weights were given by gradients of the class score with respect to each feature map.²⁴ The generated Grad-CAM (evaluation dataset) were visually inspected by a human observer, to identify regions of the image (if any) with a strong influence in the CNN decision.

3.1.

Dependence of the Classification Performance on CNN Architecture

Table 2 shows the precision and recall mean values obtained from 10 repetitions for the three CNN architectures. In this case, the models were trained using 90 deg projections [i.e., antero-posterior (AP) orientation for a typical head-first supine patient positioning], with a 75%/25% split between training and testing/evaluation datasets. These results show that the highest precision and recall averaged over all classes were achieved with VGG-16. Results from other view angles similarly showed VGG-16 had the highest performance. Therefore, only VGG-16 is considered hereafter. The head and neck classes showed high precision values regardless of the CNN architecture.

Table 2

Classification precision and recall for 90 deg projection images averaged over 10 repetitions per anatomic class for three CNN architectures using a 75%/25% split between training and testing/evaluation data. Error estimates correspond to standard deviation values, except for the class-mean, for which error estimates correspond to the square sum of standard deviation values from all classes.

3.2.

Dependence of the Classification Performance on X-Ray Projection Angle

Next, the ability of VGG16 to classify the projections from different projection angles at 45 deg intervals was evaluated by independently retraining and testing the network as described in the preceding section. Figure 2 shows precision (magenta) and recall (cyan) mean values obtained from 10 repetitions as polar plots at various projection angles. The dataset size, normalized to the maximum number of images per anatomic class, is also displayed in yellow to demonstrate the angular symmetry of the data. The highest precision ranged from 1.00 to 0.76 across classes, whereas the lowest precision ranged from 0.93 to 0.66. As for recall, the highest and lowest values ranged from 0.98 to 0.81 and 0.87 to 0.66, respectively. In most cases, the highest precision/recall values were obtained for 90 deg or 135 deg. Even though the head and neck dataset sizes were angularly asymmetric due to the short-scan CBCT acquisition, the classification performance for these classes does not reflect this asymmetry, which suggests that the performance of the model was not data limited.

Fig. 2

Mean classification precision (magenta) and recall (cyan) per anatomic class as a function of projection angle calculated using the VGG-16 CNN architecture and a 75%/25% split between training and testing/evaluation. Mean precision and recall ranges are provided below each class subplot. The training dataset size (yellow) was normalized to the maximum number of images per anatomic class. Standard deviation values are between 0.01 and 0.06 for precision and between 0.04 and 0.12 for recall.

3.3.

Dependence of the Classification Performance on Training Dataset Size

The final factor considered in this study was the size of the training dataset. The VGG-16 CNN architecture was used with 90 deg projection images and the training data set size was downsampled as described in Sec. 2.4. Figure 3 shows precision and recall mean values obtained from 10 repetitions for different training dataset sizes and demonstrates that the classification performance plateaus around 750 images ($\sim 150$ images per class), which corresponds to 50% of the original size of the training dataset. These results suggest that, for this classification task and DL model, the dataset size was sufficient for the analysis described in the preceding sections and that this architecture could be used with a smaller dataset without compromising the overall classification performance. Also, note that for head and neck classes, the plateau was reached even faster and with a higher precision, which is consistent with the robustness reported for these classes in the previous subsections.

Fig. 3

Mean classification precision (blue) and recall (orange) per anatomic class at 90 deg using VGG-16 as the training dataset size was reduced from the original 75%, and the evaluation and testing datasets sizes were kept constant. Error bars correspond to standard deviation values, except for the class-mean plot, for which error bars correspond to the square sum of standard deviation values from all classes.

3.4.

Overall Classification Performance

From Secs. 3.1 to 3.3, it was observed that one of the best performing combinations was the VGG-16 architecture trained with 90 deg projection images. In addition, it was shown that the training dataset size could be reduced to about 50% of the size of the initial training dataset, without compromising the classification performance. Table 3 shows the confusion matrix for this model implementation, assessed only on the evaluation datasets, using a 50%/50% split between training and testing/evaluation data. However, as discussed in Sec. 3.2, other projection angles achieve similar classification performance, which means that high precision/recall values were not limited to 90 deg projection images.

Table 3

Confusion matrix for VGG-16 CNN architecture trained with 90 deg (AP projection). Each cell contains the number of images categorized into each class and its corresponding percentage, relative to the number of images of a given class (zero values omitted for clarity). In this case, 50% of the data were used for training, while the remaining 50% were equally split into evaluation and testing.

From Table 3, it can be observed that the classifier has high sensitivity ($\ge 91\%$) for head, neck, and thorax but was lower for abdomen and pelvis classes. Note that, misclassified images were most frequently confounded with a neighboring class, e.g., abdomen was misclassified most frequently as thorax and pelvis as abdomen.

3.5.

Visualization of Features Identified by CNNs

Figure 4 shows representative Grad-CAMs overlayed on their corresponding 90 deg projections. Two examples are shown for each anatomic class: one correctly classified and the other incorrectly classified by the DL model summarized in Sec. 3.4, with an exception in the head class for which all cases in the evaluation dataset were correctly classified. For each image, the two most probable classes predicted by the DL model are indicated on the image along with numerical probability values output by the CNN. These images demonstrate that the activation for correctly classified cases was more heavily weighted for anatomically relevant regions, except for abdomen, where the object of focus was not entirely clear. In the case of head, we identified a higher emphasis on the cranial bones. For neck class, high weights were located close to the mandible and clavicle. For thorax, the highest weights were in the lung. For abdomen, the high weights do not seem to have a correlated landmark in the patient but were located at the borders of the body habitus. This was also true for pelvis, but in this case a high emphasis was additionally identified close to the hip joint. These trends were consistent for most of the reviewed cases (other examples are provided as supplementary material) and suggest that the CNN was learning anatomic features to perform the classification correctly. In general, the CNN did not fixate on image artifacts, metal implants (e.g., hip prostheses), or medical devices (e.g., pacemakers). In some projections, high weights were located at the inferior border of the images as shown by the red bands in some of the images in Fig. 4.

Fig. 4

Representative cases of correctly classified and misclassified 90 deg projection images for each anatomic class; all head projections were correctly classified. The Grad-CAM is displayed in a normalized “jet” color scale, and the projection image is displayed in an 8-bit grayscale. Reference boney structures for abdomen and pelvis images are indicated with dashed lines, for ease for visualization. The two most probable classes predicted by the CNN are indicated with corresponding numerical probability values, in the upper left-hand corner of each image. For this model, VGG-16 was used along with a 50%/50% split of the data for training and testing/evaluation.

Another interesting feature of Grad-CAM was the weight distribution for misclassified cases. Even though anatomic structures, such as cranial bones or lungs, were also emphasized in misclassified cases, such emphasis does not match with that of correctly classified cases. In other words, while the CNN was still successful in identifying key structures, the network’s attention was shifted to a nearby anatomic landmark (i.e., lung identified in an abdomen projection).