https://orcid.org/0009-0008-4924-7617
Figures
Abstract
Objective
Protective color restoration of faded digital pathology images based on color transfer algorithm.
Methods
Twenty fresh tissue samples of invasive breast cancer from the pathology department of Qingdao Central Hospital in 2021 were screened. After HE staining, HE stained sections were irradiated with sunlight to simulate natural fading, and every 7 days was a fading cycle, and a total of 8 cycles were experienced. At the end of each cycle, the sections were digitally scanned to retain clear images, and the color changes of the sections during the fading process were recorded. The color transfer algorithm was applied to restore the color of the faded images; Adobe Lightroom Classic software presented the histogram of the image color distribution; UNet++ cell recognition segmentation model was used to identify the color restored images; Natural Image Quality Evaluator (NIQE), Information Entropy (Entropy), and Average Gradient (AG) were applied to evaluate the quality of the restored images.
Citation: Sun G, Yan X, Wang H, Li F, Yang R, Xu J, et al. (2023) Color restoration based on digital pathology image. PLoS ONE 18(6): e0287704. https://doi.org/10.1371/journal.pone.0287704
Editor: Diego Oliva, Universidad de Guadalajara, MEXICO
Received: October 28, 2022; Accepted: June 9, 2023; Published: June 28, 2023
Copyright: © 2023 Sun et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data has been uploaded to Dryad. breast cancer fading pathological images dataset, Dryad, Dataset, https://doi.org/10.5061/dryad.0cfxpnw7b.
Funding: Jing Xu was supported by the Chinese Society of Clinical Oncology (CSCO) (No. Y-2019AZQN-1078).URL:http://www.csco.ac.cn/.The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Hematoxylin-eosin staining (HE) is a routine staining method for pathological sections, which can better show the tissue structure and cell morphology and can be used to observe and describe the morphology of normal and diseased tissues. Pathology sections have two general uses: clinical pathology diagnosis and pathology teaching. H&E-stained slides carry pathological information about the patient and provide a reliable basis for diagnosing the disease and guiding subsequent treatment. It is estimated that there will be about 19.2 million new cases of cancer worldwide in 2020, an increase of 1.1 million from 18.1 million in 2018 [1]. The biological information contained in the pathological section is an important basis for the pathologist to make a diagnosis. It is roughly estimated that each tumor patient will produce 5 pathological slides during the diagnosis process, so at least 96 million pathological slides will be produced each year. In addition to the diagnostic role, pathological sections are also an important basis for differentiation [2–5]. Each section carries unique pathological information, however, if not preserved properly, the original HE-stained sections will fade or even completely fade over time, resulting in the loss of valuable pathological information. Therefore, it is essential to restore the pathological information in these faded sections.
The concept of color transfer was first proposed by Reinhard in 2001 [6]. Color transfer refers to the process of transferring the color of the target image to the original image to obtain a color-repaired image. Image enhancement algorithms have been widely used in image restoration tasks [7, 8]. The color transfer algorithm belongs to a means of image enhancement, which is an important way to change the expression of image information in image processing. This method is suitable for scenarios where both the reference image and the input image have a single-color distribution. When the color distribution in the image is rich, a comparison between the resulting image obtained by Reinhard’s algorithm and the original input image reveals that the color transfer does not achieve the purpose of changing the overall color balance of the image. Later Welsh proposed a remap grayscale image transfer algorithm based on the color transfer algorithm [9]. Grayscale images have only luminance information, so the algorithm achieves automatic colorization of grayscale images mainly by matching the luminance values of pixels. In this paper, we aim to explore a restoration method based on faded digital pathology images, which can avoid the tedious re-staining process, protect the tissue sections from damage by the re-staining process, and restore the red-blue contrast and original pathological features of the sections.
Information and methods
HE staining and light fading of pathological sections
Twenty fresh tissue samples of invasive breast cancer were collected in 2021, all from female patients aged≥50 years with a mean age of (54.63±4.36) years. The lesion tissues were fixed and hardened using the paraffin embedding method, cut into 4-μm slices using a microtome, and the wax slices were then picked up with forceps and laid flat on the surface of water at 40–45°C. The wax strips were left to spread naturally, and the slices were retrieved and dried before being fully automatically stained using a Roche Ventana HE600 staining machine. Sections with a score greater than 90 were included in the study according to the scoring criteria outlined in Table 1 [10].
After HE staining, the sections were irradiated with sunlight to simulate natural fading. Each fading cycle lasted for 7 days, and a total of 8 cycles were completed. At the end of each cycle, the sections were digitally scanned to retain clear images, and the color changes of the sections during the fading process were recorded. Sunlight was chosen to discolor the slices because this method is simple, does not require opening the closed slices, avoids damage to the slices, and is more in line with the natural discoloration process of the slices. All slices are scanned using the Motic EasyScanner HD pathology section scanner, which is equipped with a fully automatic scanning platform: main camera: 2/3-inch CCD chip; resolution: 2448×2048; focus camera: CCD chip, resolution≥1360×1024; macro camera: CCD chip, resolution: 2048×1536. There are two digital image resolutions to choose from when scanning: 20X: 0.24um/pixel, and 40X: 0.12um/pixel, and we choose to scan the image at 0.12um/pixel resolution. We finally screened HE pathology images from 6 patients and selected 12 ROI regions from them for study, with a size of 672×672 pixels, and intercepted the corresponding ROI region at the same position of the faded image in each cycle to facilitate our color control, (Fig 1).
Sections with a score greater than 90 were included in the study. After HE staining, HE stained sections were irradiated with sunlight to simulate natural fading, and every 7 days was a fading cycle, and a total of 8 cycles were experienced. At the end of each cycle, the sections were digitally scanned to retain clear images, and the color changes of the sections during the fading process were recorded.
Color transfer algorithm and cell recognition segmentation model
The color transfer algorithm is borrowed from Reinhard’s algorithm, which means that the transferred image acquires the colors target image while preserving the structure of source image.
The basic idea is to determine a linear transformation based on the statistical analysis of the colored image, so that the target image and the source image are in laβ having the same mean and variance in space. Assuming l, a, and b are the source images lαβ. The original data of the channel, L, A, and B are the transformed images obtained after transformation lαβ. The values of the channels, ml, ma, mb and ml’, ma’, mb’ are the mean values of the three color channels of the source and target images, respectively. nl, na, nb and nl’, na’, nb’ represent their standard deviations. The formulas for transformation are:
We also use prior information to classify the image into multi-core regions, few-core regions and no-core regions. Furthermore, to enhance the contrast between the nucleus and the cytoplasm, the algorithm incorporates a contrast enhancement algorithm to improve the visual quality of the slice (Fig 2). UNet++ model was chosen to segment the cell and to verify whether the restored images can be used for intelligent diagnosis as well [11–13].
Calculate the mean and standard deviation of two images, and determine a linear transformation based on the statistical analysis of the colored image, so that the target image and source image are in a linear transformation la13 having the same mean and variance in space.
Image evaluation
NIQE (Natural Image Quality Evaluator) [14], Entropy (Information Entropy) [15], and AG (Average Gradient) [16] were applied to evaluate the image quality before and after restoration. NIQE is a collection of "quality-aware" statistical features based on the natural scene statistics (NSS) model in the spatial domain, which can evaluate the image quality without a reference image. The smaller the NIQE value, the higher the image quality. AG can be used to measure the clarity of an image, and the higher the value, the clearer the image.
Statistical treatment
SPSS 26.0 software was applied for statistical analysis. The measurement data were expressed as mean ± standard deviation or median quartiles, and the comparison of image quality before and after restoration was based on whether the difference conformed to a normal distribution using the t-test for two paired samples and the signed rank-sum test for paired data, respectively. P < 0.05 indicated that the difference was statistically significant.
Results
Faded images and RGB curves
Hematoxylin-eosin staining is both a standard stain for human histological examination and reveals cellular details. After staining the tissue with hematoxylin-eosin, the nucleus is blue, and the cytoplasm and extracellular matrix are pink. The images are brightly colored and contrasting before the start of the fading cycle, allowing the intrinsic structure of the cells to be observed. At the end of the first fading cycle, the cytoplasm and extracellular matrix fade significantly, and eosin is light-sensitive and easily fades in the case of closed sections. The color of the nucleus changed little and gradually turned pale pink with the growth of the fading cycles. The faded images make disease diagnosis difficult. (a) is the section image after automatic staining, (b)-(i) are the scanned images of sections after 1~8 fading cycles, and the fading curves are plotted based on the RGB values of the images (Fig 3).
A: Color change of pathology sections after light treatment: (a) digital pathology images after fully automated staining by Roche Ventana HE600 staining machine. (b)-(i)digital scanned images at the end of each fading cycle. B: Folding graph of RGB value change of fading images.
Image visual contrast and RGB histogram
The faded image was restored with color transfer and contrast enhancement algorithms, and the restored image restored the pathology image color to clearly observe the cell structure, and after two pathologists judged that the restored pathology image color and cell structure could meet the diagnostic needs. We also applied the RGB histogram to objectively and directly evaluate the image color information by displaying the R channel, G channel, and B channel information in the histogram, respectively. RGB histograms are red, green, and blue histograms overlapped together, where the overlapping parts will show the corresponding color. The part of the red histogram overlapped with the green histogram is yellow, the part of the blue histogram overlapped with the green histogram is cyan, the part of the red histogram overlapped with the blue histogram is magenta, and gray-green is the part of all three histograms overlapped together. In the histogram pre-faded images with bright color, the R, G, and B channels partially overlapped. After fading, the image color was single overall pink and lacked red and blue contrast, so the R, G, and B channels in the histogram were highly overlapped with gray color. The RGB histogram of the repaired image shows that the overlapping area of the three color channels is reduced, which is closer to the distribution structure of the histogram before the fading (Fig 4).
Image quality evaluation
We first conducted a visual comparison of the repaired images. Compared to Photoshop, our method is more natural in color restoration and highlights the details of cell structure (Fig 5). The objective evaluation indicators of the image also show that color restoration significantly improves the quality of the image.
Compared with the faded image (B), the restored image (C) shows a significant decrease in NIQE and an increase in Entropy and AG, all of which have statistical differences. It is worth noting that there is no statistical difference in NIQE and Entropy between the restored image (C) and the HE-stained image (A). NIQE can evaluate the image quality of super-resolution images without a reference image, and a smaller value indicates better image quality. Entropy represents the rich information contained in the image. The larger the value, the more information the image contains and the better the image quality. The AG represents the sharpness of the image, and a larger value indicates a clearer image. The sharpness of the faded image is significantly reduced, while the restored image is enhanced by the contrast enhancement algorithm. Therefore, the clarity of the restored image is significantly improved, even exceeding that of the faded image. The color transfer and enhancement algorithm effectively restored the color and information of the faded image, and there is no significant difference in both NIQE and Entropy between the restored image and the HE-stained image (Table 2).
Cell recognition results with an artificial intelligence algorithm
In our breast cancer pathological image classification task, we utilized ResNet101 as the backbone architecture and processed the images at their original size of 672 × 672 pixels. Given the unique characteristics of breast cancer pathology images, we developed a Discrimination Enhancement Module (DEM) that enhances the margin between different classes in the feature space. We achieved this by applying channel attention and spatial attention techniques to highlight the features of the foreground object in all of the channels. Specifically, the channel attention mechanism selectively emphasizes informative features in each channel by applying a global pooling operation followed by a fully connected layer and a sigmoid activation. Meanwhile, the spatial attention mechanism identifies the most relevant regions in the image by performing convolutional operations across the spatial dimensions, and then applying a softmax activation to obtain a spatial attention map. By combining these two techniques, we were able to improve the distinction among different classes and achieve higher classification performance in our breast cancer pathological image classification task.
We selected 8 regions rich in tumor cells for counting, and applying the enhancement algorithm to the pre-faded images alone can improve cell recognition to some extent. The faded images were very poor in cell recognition because of the loss of color contrast, and there was even no way to recognize the cells in some of the images. After applying the color transfer and contrast enhancement algorithm, the image quality and visual effect are effectively restored. The cell recognition rate of the restored image is basically the same as the cell recognition rate before fading. The restored images can meet the requirements of both visual and deep learning, and meet the needs of multiple uses of images (Fig 6).
Discussion
Survival of tumor patients is significantly longer with the improvement of medical treatment. However, some patients may develop new tumors in other parts of the body during their survival period. To clarify whether the newly discovered tumor is a primary tumor in that organ or whether the previous tumor has metastasized, it is necessary to compare the tissue section of the newly discovered tumor with the pathological section of the previous one. However, due to the light and humidity of the storage environment, HE staining of pathology sections faded. Faded pathology sections pose difficulties in diagnostic and differential diagnosis work. Previously, the main method to solve the problem of discoloration of pathology sections in clinical work was to stain again, and the main process included: first, the discolored sections were soaked in xylene for 72h and then the coverslips were removed, and then it was necessary to continue to soak in xylene for 24h to remove the residual gum on the tissue surface, then soak in anhydrous ethanol for 2min, soak in 95% ethanol for 1min, rinse with water, then soak in citric acid solution for 15min, and finally rinse with water and then stain the tissue slides with hematoxylin-eosin [17]. Pathology section re-staining technique can recolor the tissue, but there is a risk of dislodging the tissue section during the removal of the coverslip and the process takes a long time, in addition to the molecular structure of the discolored tissue changes resulting in uneven tissue coloration affecting the diagnosis. Even if the faded sections are re-stained, permanent preservation is still not possible. Applying color transfer algorithms to repair pathology sections avoids damage to section parenchyma, reduces the use of chemical reagents, and reduces the risk and danger of occupational exposure to chemical reagents for pathologists. Applying color transfer algorithm to repair a faded digital slice only takes 30 mins, The restored digital images no longer face fading problems and can be stored permanently, enabling the accumulation of pathological images big data, increasing the number of rare pathology images and enriching the variety of pathology teaching images.
Digital pathological images are different from ordinary images. On the one hand, each digital pathological image contains billions of pixels, which is significantly larger than ordinary images. On the other hand, digital pathology images also contain cell level fine structures. Existing image processing software cannot process the entire digital pathology images, and the color normalization algorithm proposed by Zarella and Janowczyk et al. [18, 19]. has the problem of over dependence on the original image and information loss of the processed image. Therefore, the difficulty of processing huge images and restoring the image color while also effectively preserving the fine structure of the image is the difficulty and focus of digital pathology image restoration. To solve this problem, we restored the images of faded slices by dividing the whole image into many patches, and then dividing the patches into multi-core regions, few-core regions and no-core regions according to the number of nuclei contained in the image. The RGB value characteristics of these three areas are counted, and template matching is performed based on the RGB value information, and then the faded image is restored. This solves the problem of difficult computer processing of overly large digital pathology images and avoids the image distortion problems associated with using a single template. Our image restoration algorithm effectively restores the color of pathology images, while also applying contrast enhancement algorithms to significantly improve image quality.
Digital pathological images are widely used. Apart from artificial diagnosis and pathological teaching, they are most widely used in the field of artificial intelligence diagnosis. The realization of artificial intelligence diagnosis cannot be separated from the recognition and segmentation of cells by network models. At present, the network models with good segmentation effects for pathological images mainly include VGG, ResNet, GoogLeNet, DenseNet and UNet++ [20–24]. The excellent results shown by the network model are inseparable from the good image quality. In order to verify whether the quality of our restoration image meets the requirements of the deep learning network model, we choose the UNet++ model to recognize the pathology images cells for segmentation, and the results show that we repair pathological images with excellent cell recognition results. We also note that applying the contrast enhancement algorithm alone on top of the original image improves the cell recognition rate, which suggests the need to pre-process the image before the image study, as this can enhance our experimental results. Because of the differences between pathology image datasets, some scholars normalized the images to reduce the differences between them. Color normalization of digital pathology images can effectively improve the visual quality of the images but they only adjusted for color [25–28]. Changing the color while increasing the contrast to highlight structural details can enhance the model results more effectively. Applying multi-stage detection methods and using multiple thresholds to extract texture information at different levels for clustering pathological images to improve cell detection rate [29, 30].
Conclusion
Limitations of this study: only breast cancer pathology sections were repaired, and this method can be extended to other tumor sections. In summary, the application of a color transfer algorithm for color recovery of faded sections can restore the red-blue contrast of the sections, which meets the requirements of medical pathology images and facilitates retrospective evaluation by pathologists. We should actively promote the digitization process of pathology slides and realize digital permanent preservation.
References
- 1. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin. 2021;71(3):209–249. pmid:33538338.
- 2. Cardoso F, Kyriakides S, Ohno S, Penault-Llorca F, Poortmans P, Rubio IT, et al; ESMO Guidelines Committee. Electronic address: clinicalguidelines@esmo.org. Early breast cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up†. Ann Oncol. 2019;30(8):1194–1220. pmid:31161190.
- 3. Feldman AT, Wolfe D. Tissue processing and hematoxylin and eosin staining. Methods Mol Biol. 2014;1180:31–43. pmid:25015141.
- 4. Javaeed A, Qamar S, Ali S, Mustafa MAT, Nusrat A, Ghauri SK. Histological Stains in the Past, Present, and Future. Cureus. 2021;13(10):e18486. pmid:34754648.
- 5. Chan JK. The wonderful colors of the hematoxylin-eosin stain in diagnostic surgical pathology. Int J Surg Pathol. 2014, 22(1):12–32. pmid:24406626.
- 6. Reinhard E. Aishikhmin M, Goch B, et al. Color transfer between images. IEEE Computer Graphics and Applications. 2001, 21(5):34–40.
- 7. Zhang W, Zhuang P, Sun H, Li G, Kwong S, Li C. Underwater Image Enhancement via Minimal Color Loss and Locally Adaptive Contrast Enhancement. IEEE Trans Image Process. 2022;PP:10.1109/TIP.2022.3177129. pmid:35657839.
- 8. Zhuang P, Wu J, Porikli F, Li C. Underwater Image Enhancement With Hyper-Laplacian Reflectance Priors. IEEE Trans Image Process. 2022;31:5442–5455. pmid:35947571.
- 9.
Inverardi P, Tivoli M. Software Architecture for Correct Components Assembly[C]// Formal Methods for Software Architec2 tures, vol. 2804 of Lecture Notes in Computer Science, Springer, 2003: 922121.
- 10.
Chinese Medical Association. Clinical Technique Practice Specification-Pathology Sub-volume. Beijing People’s Military Medical Publisher, 2004:6–7.
- 11. He H, Zhang C, Chen J, Geng R, Chen L, Liang Y, et al. A Hybrid-Attention Nested UNet for Nuclear Segmentation in Histopathological Images. Front Mol Biosci. 2021, 8:614174. PMCID: PMC7925890. pmid:33681291
- 12. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation. IEEE Trans Med Imaging. 2020, 39(6):1856–1867. PMCID: PMC7357299. pmid:31841402
- 13. Du XF, Wang JS, Sun WZ. UNet retinal blood vessel segmentation algorithm based on improved pyramid pooling method and attention mechanism. Phys Med Biol. 2021,66(17): https://doi.org/10.1088/1361-6560/ac1c4c pmid:34375955.
- 14. Wu L, Zhang X, Chen H, et al. VP-NIQE: An Opinion-Unaware Visual Perception Natural Image Quality Evaluator. Neurocomputing, 2021, 463(5).
- 15. Zhuang L, Guan Y. Adaptive Image Enhancement Using Entropy-Based Subhistogram Equalization. Comput Intell Neurosci. 2018, 2018:3837275. pmid:30186315.
- 16. He K, Zhang X, Ren S, et al. Deep residual learning forimage recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:770–778.
- 17. Zhiling Qu, Juan Ni, Qilin Ao, et al. Discussion on color restoration technology of HE slices in the teaching of old fading pathology. Journal of Clinical and Experimental Pathology, 2011, 27 (04): 433–434
- 18. Zarella MD, Yeoh C, Breen DE, Garcia FU. An alternative reference space for H&E color normalization. PLoS One. 2017;12(3):e0174489. https://doi.org/10.1371/journal.pone.0174489 PMCID: PMC5371320.
- 19. Janowczyk A, Basavanhally A, Madabhushi A. Stain Normalization using Sparse AutoEncoders (StaNoSA): Application to digital pathology. Comput Med Imaging Graph. 2017;57:50–61. pmid:27373749.
- 20. Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking theinception architecture for computer vision[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:2818–2826.
- 21. Huang G, Liu Z, Maaten L V D, et al. Densely connected convolutional networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:4700–4708.
- 22.
Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-assisted Intervention. Springer, 2015, 234–241.
- 23. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40:834–848. pmid:28463186.
- 24. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. UNet++: A Nested U-Net Architecture for Medical Image Segmentation. Deep Learn Med Image Anal Multimodal Learn Clin Decis Support (2018). 2018, 11045:3–11. PMCID: PMC7329239. pmid:32613207
- 25. Roy S, Kumar Jain A, Lal S, Kini J. A study about color normalization methods for histopathology images. Micron. 2018, 114:42–61. pmid:30096632.
- 26. Onder D, Zengin S, Sarioglu S. A review on color normalization and color deconvolution methods in histopathology. Appl Immunohistochem Mol Morphol. 2014, 22(10):713–719. pmid:24897076.
- 27. Tosta TAA, de Faria PR, Neves LA, do Nascimento MZ. Color normalization of faded H&E-stained histological images using spectral matching. Comput Biol Med. 2019, 111:103344. pmid:31279982.
- 28. Moradmand H, Aghamiri SMR, Ghaderi R. Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma. J Appl Clin Med Phys. 2020 Jan;21(1):179–190. PMCID: PMC6964771. pmid:31880401
- 29. Fekri-Ershad Shervan. Cell phenotype classification using multi threshold uniform local ternary patterns in fluorescence microscope images. Multimedia Tools and Applications. (2021) 80: 12103–12116. https://doi.org/10.1007/s11042-020-10321-w
- 30. Shafiei Farzaneh, Fekri-Ershad Shervan. Detection of Lung Cancer Tumor in CT Scan Images Using Novel Combination of Super Pixel and Active Contour Algorithms. Traitement Du Signal. 2020, pp. 1029–1035. https://doi.org/10.18280/ts.370615