2.1.

Histologic Slides from Head and Neck SCC Patients

Twenty-six H&E-stained histologic slides were obtained from larynx, hypopharynx, buccal mucosa, and floor of mouth (FOM) of 20 different head and neck SCC (HPV-negative) patients, as we previously described.^7,31 There were three types of slides, namely T, N, and TN. The tissue of each TN slide was resected at the tumor-normal margin, containing both cancerous and normal tissues, while a T slide contains only cancerous tissue, and an N slide is normal tissue. The TN slide was first selected for each patient, but for six patients whose TN slide did not contain enough cancerous or normal tissue, a T slide or N slide would be included to balance the data. All slides were previously digitized with $40\times$ objective magnification and manually annotated by a board-certificated pathologist. In this work, we used the annotated cancerous and normal regions in the digital histology images as ground truth for data selection and classification.

In this study, we manually selected regions of interest (ROIs) for both types of tissue for hyperspectral image acquisitions. For each slide, we chose at least three ROIs in the cancerous region annotated by the pathologist who specialized in HNC. We also selected at least three ROIs for the normal tissue. Considering the potential interpathologist variation of the histological diagnosis, both cancerous and normal ROIs were away from the edge of the annotation to avoid any interface area. In addition, there should be adequate nuclei in the images for extraction and classification; therefore, the selected cancerous ROIs were at or close to the cancer nests, where a mass of cells extends to the surrounding area of cancerous growth. The selected normal ROIs were from the healthy stratified squamous epithelium that is far away from the cancerous regions. To make the cancerous nuclei and normal nuclei comparable, we only extracted normal nuclei from the second and third layers of the stratified squamous epithelium, from which the SCC cells originally arise. On average, over 750 nuclei were extracted for each patient, including cancerous and normal nuclei. In total, we collected 257 hyperspectral images (119 normal and 136 cancerous), from which around 15,000 nuclei were later extracted. Figure 2 shows the synthesized RGB images from the hyperspectral images of some representative cancerous and normal ROIs.

Fig. 2

Synthesized RGB images from HSI, which show some representative regions selected for nuclei extraction and quantitative testing. (a)–(d) Normal regions centered at the second and third layers of the healthy stratified squamous epithelium. (e)–(h) Cancerous regions at or close to cancer nests.

2.2.

Image Acquisition of Histologic Slides

The histological slides were scanned using our custom-made hyperspectral microscopic imaging system, which contains a transmitted-light microscope, a hyperspectral camera, and a color camera, as has been reported in our previous study.^23,27 The HSI camera acquired hyperspectral images with 87 bands in the wavelength range from 470 to 720 nm. A color camera was added to the system via an additional optical port on the microscope. The hyperspectral camera and the color camera were tuned to be par-focal, and they shared the same field of view (FOV) with slight differences around the edges due to the different sensor sizes. The light path of the microscope is shown in Fig. 3(a). The spatial size of the hyperspectral images was $2048\times 2048\text{pixels}$ while that of the RGB images was $3072\times 2048\text{pixels}$. The FOV of the hyperspectral camera under a $40\times$ magnification was about $285\mu \mathrm{m}\times 285\mu \mathrm{m}$, with a spatial resolution of $139\mathrm{nm}/\text{pixel}$. Two cameras were synchronized, so that both hyperspectral images and RGB images of the same region were acquired altogether along with the scanning of the slides.

Fig. 3

Image acquisition and registration. (a) Diagram of the microscope light path. (b) System calibration by registering the hyperspectral image and the RGB image of the microscopic calibration target. (c) Registering each pair of hyperspectral image and RGB image of the slide by applying the previously obtained registration matrix on the RGB image.

Before acquiring images of the histologic slides, we first implemented system calibration by imaging a microscopic calibration target, as shown in Fig. 3(b). Both the hyperspectral image and RGB image were transferred to grayscale images, and then the RGB-synthesized grayscale image was registered to the HSI-synthesized grayscale image using affine registration with the oriented FAST and rotated BRIEF feature detector³² from the OpenCV2 package. The obtained registration matrix was saved as the system calibration matrix, and for each pair of hyperspectral and RGB images that was acquired during slide scanning, we registered the RGB image to the matching hyperspectral image using this matrix, as shown in Fig. 3(c). Note that the “HSI” images in Figs. 3(b)–3(c) are RGB images synthesized from hyperspectral images, which will be explained in the next section.

Before the image acquisition, a blank area on the slide without any tissue or dust was selected. Under the same illumination and focusing setting as during the slide scanning, we first set the white balance of the color camera based on this area, then a hyperspectral image of the same region was taken and used as the white reference image. Dark balance of the color camera was set by blocking the objective lens, and the hyperspectral dark reference image was acquired automatically by the hyperspectral camera along with the acquisition of the tissue images.

Because our hyperspectral microscopic imaging system is based on transmitted-light microscopy, the pixel values in the acquired images are related to the intensity of the transmitted light from the histologic slides. All hyperspectral images of the tissue were calibrated with the white reference image and dark reference image to obtain the normalized transmittance of the slides, as follows:

To visualize the selected ROIs, we synthesized RGB images from the hyperspectral data using our customized transformation function, which was made up of multiple cosine functions, as shown in Fig. 4(a). We developed it based on the spectral response of human eye perception to colors,³³ but we modified the channel weights for red (R), green (G), and blue (B) due to the absence of the wavelength bands in 380 to 470 nm in our hyperspectral camera. To compensate the absent wavelength bands and balance the colors, we increased the channel weights of red within 380 to 500 nm and the channel weights of blue within 380 to 550 nm. In addition, the colors that we see through our eyes are influenced by the human eye color perception and the spectral signature of the light source. However, the image calibration in Eq. (1) has removed the influence of light source from the hyperspectral images. Therefore, to have the color of the synthesized RGB images as real as possible, we took the light source into consideration and slightly increased the channel weight of the red region from 500 to 720 nm, because the irradiance of the halogen light source is higher in the red-color wavelength range. After applying this customized transformation, the synthesized RGB image was multiplied with a constant of two to adjust the brightness. The synthesized RGB images offer higher contrast and clear visualization of the cellular structures than a single band within the hyperspectral image; and the color was close to the original RGB histologic images, as shown in Figs. 4(b) and 4(c).

Fig. 4

Synthesize RGB image from the hyperspectral data. (a) Customized transformation function for synthesizing an RGB image from a hyperspectral cube, modified from the human eye spectral response for red (R), green (G), and blue (B). The gray area shows the unavailable wavelength range of our hyperspectral camera. (b) A hyperspectral cube and its synthesized RGB image using the transformation in (a).

2.3.

Automatic Nuclei Segmentation

SCC nuclei exhibit various cancer-related information. Therefore, by extracting nuclei from the whole slide and implementing cancer detection based on nuclei classification, we can potentially avoid the redundancy of information. Here, we propose a nuclei segmentation method based on PCA. PCA is a multivariate technique for spectral data analysis. It projects the high-dimensional spectral data into a lower-dimensional space and removes the correlation among the wavelength bands. Therefore, it is able to extract the principal component images that have the highest variance, in other words, the highest contrast.^34,35 Due to the fact that nuclei are stained by hematoxylin and cytoplasm is stained by eosin,³⁶ the different optical absorption properties of the two stains can be used by PCA to form distinctive principal components, where the contrast between nuclei and cytoplasm may be increased.

To implementing PCA on the hyperspectral data, each 87-band hyperspectral image was reshaped to a two-dimensional (2D) matrix with 87 vectors. Then, PCA calculation was carried out on the matrix in MATLAB^® (MathWorks Inc., Massachusetts, United States). Because of the spectral distinction among the nuclei, cytoplasm, and background, the top three principal components (PCs) highlight these three parts separately, as shown in Figs. 5(a)–5(c). Afterward, we normalized PC1, PC2, and PC3 with their maximum and minimum values

Fig. 5

PCA-based nuclei segmentation. (a) The first PC after image normalization (PC1_norm), where pixels of nuclei have lower values than those of cytoplasm and background. (b) The second PC after normalization (PC2_norm), where nuclei pixels have high values than those of cytoplasm and background. (c) The normalized third PC (PC3_norm) that highlights the background. (d) The difference image by subtracting PC1_norm from PC2_norm.

Although the nuclei in PC1 seem to be distinct, it is not easy to segment them with a hard threshold. Since the pixels of nuclei in PC1 have a lower value than those of cytoplasm and background, while the pixels of nuclei in PC2 have a higher value, the difference between the normalized PC2 and PC1 ($\mathrm{PC}2\_\text{norm}-\mathrm{PC}1\_\text{norm}$) yields an image with a high contrast of nuclei and other components, as shown in Fig. 5(d). Generally, the pixels of nuclei have positive values, while those of cytoplasm have negative values, with very a slight difference among various slides. Therefore, a binary mask can be easily generated with the difference image and a hard threshold to segment nuclei from the slides. Considering the general size of nuclei, the segmented components with a very small area were removed. For several overlapped nuclei, we could use a watershed algorithm^37,38 to separate them. It is worth noting that the shape of each segmented nucleus in the generated binary mask might not be very precise, but with this method, we were able to easily locate most of the nuclei and extract them from the hyperspectral images.

After applying the PCA-based nuclei segmentation method, the centroids of all segmented nuclei were identified. Because the RGB images were aligned with the hyperspectral images, the identified nucleus centroids also match the locations of nuclei in the registered RGB images. Hyperspectral and RGB patches centered at the centroids were extracted from the images. The patch size was set as $101\times 101\text{pixels}$, which was large enough to include the enlarged SCC nuclei and a few overlapped nuclei that were hard to separate. Afterward, we reviewed all extracted nucleus-centered patches and removed a few outliers that were out of focus. For any nucleus that was extracted from the top or bottom margin area of the registered RGB image, which was filled with black color, the hyperspectral patch and RGB patch of this nucleus would be removed from the dataset. Table 1 shows the final number of cancerous and normal nuclei patches that were extracted from each patient.

Table 1

Number of cancerous and normal nuclei extracted from each patient.

2.4.

Spectra-Based SVM Classification

For a straightforward investigation of the distinction ability of nuclei spectra, we obtained the average spectra of all extracted nuclei and carried out spectra-based nuclei classification using an SVM classifier. Although the PCA-based nuclei segmentation method could provide a rough binary mask that helped with the localization of nuclei in the image, the mask was not exactly accurate and might contain some cytoplasm pixels. To have precise spectral signatures of merely the nuclei, we manually and carefully delineated the margin of each nucleus in the extracted nucleus-centered patches, and the average spectrum of all pixels in each outlined nucleus was calculated, as shown in Fig. 6(a). The red curve shows the average transmittance spectrum of the outlined nucleus on the left, and the blue curve is the spectral signature of a blank area on the glass slide with no tissue, which shows an even transmittance across the whole wavelength range after the white reference calibration. The pink arrow indicates the location of the absorbance peak of eosin dye, while the blue arrow points at the absorbance peak of hematoxylin dye.³⁹ Because of the thickness variation of the tissue, the amplitudes of nuclei transmittance spectra can be different. Therefore, each average spectrum was then normalized by being divided by a constant, which was the sum of the spectrum at all wavelengths

Fig. 6

Average nuclei spectra extraction and normalization. (a) The average spectrum of a nucleus in an extracted image patch. (b) Average transmittance spectra of cancerous nuclei and normal nuclei from patient #16. The epithelium in this slide was thick and the nuclei were stained dark, which caused lower values of normal nuclei spectra. (c) Average normalized transmittance spectra of patient #16.

Figure 6(b) shows the transmittance spectra of cancerous nuclei and normal nuclei from one patient. Specifically, the epithelium tissue in this slide was relatively thick, and the normal nuclei had a very dark color probably caused by overstaining, which resulted in lower values of the transmittance spectra. However, after spectral normalization, the spectral signatures of cancerous nuclei and normal nuclei became more comparable. It could be seen that the values of cancerous nuclei spectra were higher in the wavelength range from 470 to 640 nm, and then became lower after 650 nm. The same trend of nuclei spectra was also observed in many other patients, which means that the spectral signatures of nuclei could provide some distinction.

The normalized average spectra of nuclei were used for the training and validation of an SVM classifier. The SVM was implemented with a radial basis function kernel using MATLAB. Leave-one-patient-out cross-validation was carried out, each time the spectral data from 19 patients were used for training, and the spectra from one patient for validation. Grid search was carried out for SVM hyperparameters $C$ and $g$ over the range of ${\log }_{2}C=\{-1,0,\dots ,4\}$ and ${\log }_{2}g=\{-1,0,\dots ,4\}$.

2.5.

Patch-Based CNN Classification

After the segmentation of nuclei and the generation of nucleus-centered patches, the patches were used for the training, validation, and testing of a 2D CNN. The CNN was implemented using Keras with a Tensorflow backend on a Titan XP NVIDIA GPU. It consisted of eight standard 2D convolutional layers, two max pooling, one average pooling, and two fully connected layers, as shown in Fig. 7 and Table 2. The numbers above the feature maps in Fig. 7 indicate the number of filters. The output had two classes, i.e., cancerous and normal. Except for the first convolutional layer that had a kernel size of $5\times 5$, all other convolutional layers had a kernel size of $3\times 3$. The strides of all convolutional layers were $1\times 1$, and that of the max pooling layers was $2\times 2$. All convolutional layers were initialized with the “Glorot-normal” kernel initializer. The rectified linear unit (ReLU) activation as well as a 10–30% dropout was applied following each convolutional layer. We tried using batch normalization and L2 regularization for the convolutional layers, but they did not improve the training and validation results in this study. The optimizer was Adam⁴⁰ with an initial learning rate of ${10}^{-5}$ as well as decay parameters beta_1 = 0.9 and beta_2 = 0.999. The loss function was binary cross-entropy. The network was trained with a batch size of 16 for 7 to 27 epochs, depending on when the validation accuracy stopped increasing.

Fig. 7

The 2D-CNN for patch-based nuclei classification, with eight standard 2D convolutional layers, two max pooling layers, one average pooling layer, and two fully connected layers. The input size is $101\times 101\times N$, where $N=3$ for RGB patches and $N=87$ for HSI. Numbers above each figure map show the number of filters.

Table 2

2D CNN architecture for patch-based nuclei classification.

As specified in Table 2, the same CNN architecture was used for hyperspectral and RGB patches to compare the classification performance and evaluate the usefulness of extra spectral information in the hyperspectral images. The network that was trained with hyperspectral patches, namely the HSI-CNN, had an input size of $101\times 101\times 87$, and the one with RGB patches, namely the RGB-CNN, had an input size of $101\times 101\times 3$. These two networks were tuned separately to avoid bias of optimization, although the learning rate and decay rate turned out to be the same.

All nucleus-centered image patches were split into training, validation, and testing groups to evaluate the network. The same data partition was used for both HSI-CNN and RGB-CNN. We carried out eightfold cross-validation, each time nuclei patches from 14 patients were used for training and those from two patients were used for validation. Four randomly selected patients were left out as an independent testing group, including 2068 nuclei patches extracted from 44 images. The data of one patient have never been used in the training, validation, or testing group at the same time. In addition, all training patches were four times augmented by rotating. Table 3 shows the number of images and nuclei patches that were used in each fold before data augmentation.

Table 3

Data partition for cross-validation.

For validation and testing, nucleus-centered patches were also four times augmented by rotating and reflecting. Then, the average probability of each nucleus-centered patch was calculated across all the four augmented versions of the patch. The optimal threshold levels were determined based on the receiver operator characteristic curves for validation data in different folds, and the same threshold was applied to all validation data in the same fold. Finally, the network with the best validation results was selected and tested on the testing data group.

2.6.

Image-Wise Classification

When doing whole-slide image classification for cancer detection, the appearance of suspicious cancerous nuclei in the image patches is a key factor for the network to make cancer prediction, even if the network was not trained to look for nuclei on purpose.⁷ But with the relatively low magnification (usually $10\times$), it may miss some of the morphological features and spectral information of nuclei, especially for the regions where few nuclei exist, thus leading to some misclassification. Theoretically, a region should be diagnosed as cancerous even if only one cancerous cell is found in it. Therefore, we propose to use the cancerous nuclei detection results for an image-wise cancerous region identification, which can potentially assist the diagnosis of the slide, especially in the regions where a whole-slide classification network is not able to achieve a precise decision making. Moreover, the FOV under a high magnification is small ($<0.3\mathrm{mm}\times 0.3\mathrm{mm}$ in this study), so the resolution of an image-wise classification would still meet the need for tumor margin detection.

In this study, image-wise classification was implemented based on the percentage of cancerous nuclei detected in each image. After extracting the nuclei patches using our proposed PCA-based segmentation method and carrying out patch-based nuclei classification using the CNN, the percentage of detected cancerous nuclei among all nuclei within each hyperspectral image was calculated. If the percentage of cancer nuclei in this image was above a certain threshold, the corresponding region on the histological slide would be considered as cancerous. Theoretically, a region should be taken as cancerous even if only one cancer cell exists there. However, considering the possible false positives in nuclei classification, we employed five different thresholds with relatively small values (1%, 5%, 10%, 20%, and 30%), and counted how many images in the validation data groups were correctly classified. The threshold value that resulted in the best image-wise classification among the validation data would finally be applied to the testing data. It is worth noting that in this study, the number of segmented nuclei from different images can be different, e.g., from 6 nuclei per image to near 200 per image, depending on the major histological structure in the image. For example, some images of the cancerous tissue may contain a large SCC pearl, or much more lymphocytes than SCC nuclei due to chronic inflammation, resulting in a very small number of nuclei in the images. For images of the normal tissue, thin epithelium could also result in a small number of extracted nuclei. Therefore, the threshold determined for image-wise classification based on images with different base numbers of nuclei should work for any region, even those that get misclassified due to the small quantity. We counted the number of images in the validation groups with a different number of nuclei detected from them. In total, there were four images with very few ($<10$) nuclei, 12 images with 11 to 20 nuclei, 90 images with 21 to 50 nuclei, 88 images with a decent number (51 to 100) of nuclei, and 19 images with $>100$ nuclei.

2.7.

Evaluation Metrics

Before the image acquisition, we carefully selected imaging ROIs in the histological slides according to the manual reference standard of pathologists specialized in HNC. Cancerous regions were selected from or close to the cancer nests, and normal regions were chosen from stratified squamous epitheliums far from the identified tumor-normal margin. In addition, all segmented nuclei were reviewed to make sure that nuclei extracted from cancerous regions were truly cancerous and those from the normal regions were truly normal. After the nuclei extraction, we looked through the nucleus-centered image patch dataset and removed the outliers.

In this study, we use the area under the receiver operating characteristic curve (AUC) as well as the overall accuracy, specificity, and sensitivity to evaluate the nuclei classification performance, as defined by Eqs. (4)–(6). Accuracy is defined as the ratio of the amount of correctly labeled nuclei to the total number of nuclei in the group. Specificity and sensitivity are calculated from true positive (TP), true negative (TN), false positive (FP), and false negative (FN), where positive corresponds to cancerous and negative to normal

3.1.

Nuclei Segmentation Results

With the proposed PCA-based nuclei segmentation method, binary masks of nuclei were generated, which could separate nuclei from the cytoplasm and background, as shown in Fig. 8. The Fig. 8(a) shows the masks of normal nuclei in the images of healthy epithelium, and Fig. 8(b) shows the masks of cancerous nuclei in cancerous tissues. Because of the slight spectral distinction and the small size, lymphocytes were not segmented. The results show that our proposed method can segment most nuclei in the image. Due to the thickness of the tissue slide, some nuclei at different layers might be out-of-focus and were not as distinct as the in-focus nuclei, and these blurry nuclei were hardly segmented. But this could potentially be solved if automatic focusing and image quality enhancement methods for hyperspectral microscopic imaging can be improved. Figure 8(c) shows representative cancerous and normal nuclei from 20 patients.

Fig. 8

Nuclei segmentation results. (a) and (b) Nuclei segmentation in normal tissue and cancerous tissue, respectively. The segmentation mask obtained using hyperspectral data aligned well with RGB images. Nuclei patches extracted from the top and bottom of registered RGB images were later removed. (c) HSI-synthesized RGB image patches of representative cancerous and normal nuclei were extracted from each patient.

3.2.

Nuclei Classification Results Using Only Spectral Information

The SVM classification using normalized average transmittance spectra of extracted nuclei obtained an average accuracy of 0.68, as well as 0.74 sensitivity and 0.54 specificity. Although the spectra-based classification did not reach a very high accuracy, it did show some distinction ability of the nuclei spectra. Because SCC cells originate from the stratified epithelium, where we obtained our normal nuclei, it is reasonable that the two types of nuclei have a similar shape in their spectral signatures. In addition, the shape variation, size change, and staining variation of SCC nuclei as well as the different cell mitosis stages could all introduce disparities to the spectra among different cancerous nuclei. The spectra-based classification was only carried out for a straightforward investigation of whether the spectral signatures of nuclei would change along with cell carcinogenesis. That being said, there were five patients, i.e., #3, #8, #10, #13, and #18, who got a fairly good accuracy above 0.85. Their spectral signatures of cancerous nuclei and normal nuclei all showed very obvious differences and were in line with the trend as previously shown in Fig. 6(c). In addition, two patients, i.e., #9 and #12, got extremely bad classification results with an accuracy $<0.4$. In the next section, we compare the patch-based classification results using the RGB patches (spatial and color information) and hyperspectral patches (spatial and spectral information) and see if their spectral difference brought a significant impact on classification.

3.3.

Nuclei Classification Using Spatial and Spectral Information

In this section, we compare the classification performance using hyperspectral patches and RGB patches of the segmented nuclei, as shown in Table 4. For the validation data, the CNN trained with HSI patches could distinguish SCC nuclei from normal epithelium nuclei with an average AUC of 0.93, as well as 0.89 accuracy, 0.88 sensitivity, and 0.89 specificity. The RGB patch-based CNN achieved an average AUC of 0.88 as well as 0.81 accuracy, 0.82 sensitivity, and 0.81 specificity. In half of the validation patients, HSI-CNN outperformed RGB-CNN with an increased accuracy of 3% to 6%, and in five patients (#3, #7, #8, #11, and #14), HSI significantly improved the accuracy by $>10\%$. It is worth noting that all five patients got SVM classification results no worse than the average, especially patients #3 and #8 got 0.88 accuracy. Interestingly, patients #13 and #18 got high classification accuracy using SVM (0.90), RGB-CNN (0.97 and 0.99), and HSI-CNN (0.98 and 0.99), while patient #9 got the worst performance using all three methods (0.35, 0.64, and 0.64). The other patient that had a very low SVM accuracy of 0.38 was #12, who was the only one that got lower accuracy using hyperspectral patches. It can be seen that spatial and spectral information contributes to the differentiation of SCC nuclei and normal nuclei. The morphological features of nuclei seem to play a more dominant role, but spectral signatures do bring a certain impact to the classification performance. Figure 9 shows the average spectra of cancerous and normal nuclei from four different patients, who (1) got very good classification results using SVM, RGB-CNN, and HSI-CNN, (2) got good SVM accuracy and significantly improved classification performance using HSI-CNN compared to RGB-CNN, (3) had less distinctive spectral signatures and an average-level SVM accuracy but still, HSI outperformed RGB, and (4) had abnormal spectral signatures and a very low SVM accuracy, which resulted in worse classification results when using spectral information together with spatial information.

Table 4

Patch-based CNN classification results of 20 patients using HSI patches and synthesized RGB patches.

Fig. 9

Validation performance and average spectra of cancerous nuclei and normal nuclei from different patients. (a) Receiver operating curves of eight validation folds with corresponding AUC values. (b) Average spectra of patient #13, who had good classification accuracy using the spectra-based SVM (0.91), RGB-CNN (0.97), and HSI-CNN (0.98). The spectral of two types of nuclei are clearly distinctive. (c) Average spectra of patient #8, who got good spectra-based classification accuracy of 0.88, and significant improvement patch-based classification accuracy using HSI-CNN (0.87) compared with RGB-CNN (0.76). (d) Average spectra of patient #6, where spectra-based classification had a close-to-average performance, but the spectral information in hyperspectral patches still helped improving patch-based classification results. (e) Average spectra of patient #12, who got an extremely low spectra-based classification accuracy probably due to the reversed trend of spectral signatures. This is also the only patient who got lower patch-based classification result using HSI.

For the four testing patients, HSI has achieved an average performance of 0.89 AUC, 0.82 accuracy, 0.72 sensitivity, and 0.93 specificity. The CNN trained with RGB patches achieved an average performance of 0.86 AUC, 0.74 accuracy, 0.87 sensitivity, and 0.62 specificity. Although HSI had a lower average sensitivity than RGB, its overall classification performance (accuracy) was significantly better. For patient #16, the spectral signatures of cancerous and normal nuclei were obviously differentiable, and both networks got very good results with a 0.99 AUC. Except patient #16, the classification results for the other three patients using RGB patches and hyperspectral patches were very different. For patients #4 and #19, most SCC nuclei did not show as much size change and shape variation as in other slides, hence the spatial information was insufficient for nuclei differentiation, while using the spectral information in the hyperspectral patches compensated for this insufficiency and significantly improved the results. Therefore, the HSI-CNN achieved an AUC above 0.96 and over 20% improvement in accuracy in both patients. Patient #20, on the contrary, was negatively impacted when using the spectral information, which resulted in very low accuracy of 0.63 as well as an AUC of 0.62. By looking at the average spectra, we found that patients #20 and #12 had the same issue of a reversed spectral trend. The reason that caused this change of spectral signatures was uncertain. One possible explanation is the color variation, which we will in the future improve our network to take care of. However, it is worth investigating whether specific pathological features could possibly result in this change.

3.4.

Image-Wise Classification Results

In this study, the goal of distinguishing cancerous and normal nuclei is to facilitate automatic cancer detection in the histological slides. Therefore, we carried out image-wise classification based on the percentage of cancerous nuclei detected in the image to locate cancerous tissue. For each single hyperspectral image, the number of nuclei that were classified as cancerous ($N_C$) and the total number of segmented nuclei in this image ($N_{\text{Total}}$) were calculated, and the ratio of the two numbers $N_C/N_{\text{Total}}$ was used as to identify cancerous regions. Technically, the region should be counted as cancerous if there’s even one cancerous cell in it. However, considering the false positives, we tried five different thresholds for the percentage of cancerous nuclei, namely 1%, 5%, 10%, 20%, and 30%, on the validation data. The number of correctly classified images with each threshold is shown in Table 5. Although using a larger threshold such as 20% could result in more correctly classified images, it also increases the risk of false negatives. It is worth noting that most of the misclassified images were from patients #9 and #12, where the classification performance was generally not very good. As a result, the threshold of 10% was selected and applied to the 44 testing images from four different patients. All 27 cancerous images as well as 15 normal images were correctly identified, resulting in a total accuracy of 0.95. The two normal images that were classified as cancerous were both from patient #20. The percentage of detected cancerous nuclei in these two images was both slightly above 20%. Despite the low sensitivity of patient #20, all cancerous regions were correctly labeled.

Table 5

Image-wise classification results in the validation group using different thresholds.