Integrating deep convolutional neural networks with marker-controlled watershed for overlapping nuclei segmentation in histopathology images
Introduction
Histopathology is about the investigation of the manifestations of disease based on microscopic examination of tissue structure, which has been viewed as a gold standard for the diagnosis of most cancer diseases, such as the breast cancer [1], lung cancer [2], and prostate cancer [3]. In general, the tissue specimens of patients should be stained with special colors for highlighting the interesting components of tissue such as the lymphocytes, cancer nuclei, and glands [4]. For example, hematoxylin eosin (HE) is one of the most widely used stain materials, which imparts the blue-purple and pink color to the nuclei and cytoplasm respectively (see Fig. 1). Then the pathologists would observe the characteristics of histological structure under a microscope, which is an important step for grading and diagnosis of disease [5], [6]. Nevertheless, this work is very laborious and time-consuming for pathologists. In addition, the diagnosis result is likely to be influenced by the cognitive diversity due to the lack of quantitative analysis. With the development of digital pathology technology, the image information of tissue can be captured by the whole slide imaging (WSI) scanners, stored as digital image by compression algorithm and displayed in the computer screen [7]. Furthermore, some researchers have employed image processing and artificial intelligence technology to establish computer-aided diagnosis tools [8], [9], [10] for improving the work efficiency and reducing error rates for pathologists.
Segmentation of cell nuclei is an indispensable step for automatic digitized histopathology imagery analysis system [11], [12]. Unfortunately, as stated in literatures [11], [13], [14], [15], [16], this task is difficult due to (1) the dense overlapping between nuclei, (2) the complex variability in size, shape, appearance, and texture of the individual nuclei, and (3) the non-homogenous background, as shown in Fig. 1. In addition, the difference in tissue type, cell type and staining material of histopathology images is also a great challenge to the robustness of automatic segmentation algorithms [12], [17]. In this paper, we mainly focus on the overlapping nuclei segmentation task, which is to divide the touching nuclei into several individual nucleus instances.
To solve this problem, we propose an efficient computing framework by combining marker-controlled watershed (MCW) [18] with convolutional neural networks (CNN) [19], as illustrated in Fig. 2. The main novelty and contribution of our work can be summarized as below: (1) presenting a multi-task network architecture to simultaneously learn the foreground, interval, and marker information of nuclei from the original images, (2) integrating the marker and interval results with the MCW method as a post-processing method for separating the overlapping nuclei in the foreground, and (3) proposing an efficient method to extract the mask of nuclei interval and marker from the annotation of nuclei foreground for training our network. Note that the interval of nuclei indicates the gap between adjacent nuclei, and the marker of nuclei represents the central location of each nucleus. These important prior information can greatly improve the robustness and accuracy of MCW method in touching nuclei segmentation. By contrast, Naylor et al. [17], [20] simply extracted the mark of nuclei from the output of CNN with ignoring the importance of interval information. Moreover, the errors in segmentation result would impair the accuracy of mark extraction method, and further lead to the over-segmentation and under-segmentation problems for MCW method. The comparison with state-of-the-art methods can fully prove the superiority and effectiveness of our method.
The remainder of our paper is organized as follows. In Section 2, we review some related work with overlapping nuclei segmentation. Section 3 introduces the framework of our method. The implementation details of our method are presented in Section 4. Then the experimental results are shown in Section 5. Finally, the conclusion and perspectives are drawn in Section 6.
Section snippets
Related work
Thresholding and morphological operation were two widely used nuclei segmentation methods in the early stage of this research area [21]. For examples, Gurcan et al. [22] established a nuclei segmentation framework for neuroblastoma cancer analysis by combining morphological reconstruction with hysteresis thresholding [23]. Nawandhar et al. [24] adopted the Otsu method [25] and multiple morphological operations to segment the cell nuclei in colon tissue. In addition, Huang et al. [26] proposed a
Our proposed method
In this paper, we design a novel deep interval-marker-aware network (DIMAN) with end-to-end training and pixel-wise annotation properties. In our network, the multi-scale feature maps are stacked as a feature pyramid by skip connection. Moreover, three pixel-wise classifiers are connected with the feature pyramid in parallel for learning multiple classes of objects from the original images. Then we systematically combine these learned results into MCW method to obtain the final segmented and
Datasets
In our experiments, we evaluate the performance of nuclei segmentation methods on three public histopathology image datasets:
Dataset11: the training dataset of MICCAI 2017 Digital Pathology Challenge, which includes 32 annotated HE stained histopathology image tiles of four different types of cancer. The nuclei images were extracted as rectangular tiles from the whole slide images at high resolution. Each labeled mask file is an array of
Comparison with other networks and post-processing methods
We compare our proposed approach with four known CNN-based semantic segmentation methods, including FCN8s [38], HED [39], Unet [40], and SharpMask [41]. For fairness, all competing networks were trained under the same conditions with our method. In addition, the conditional erosion based watershed (CEW [28]) method [28], morphological dynamics based watershed (MDW [17]) method [17], fitting ellipses based segmentation (FES) method [42], and concave points based segmentation (CPS) method [43]
Conclusion and future work
It’s well known that overlapping nuclei is one of the major challenges for nuclei segmentation in histopathology images. For tackling this problem, we have established a novel computational framework by combining the convolutional neural networks with marker-controlled watershed. Unlike other competing methods, which extract the markers of nuclei from the foreground segmentation result, we proposed a deep CNN architecture for predicting the foreground, interval and marker of nuclei
Declaration of Competing Interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work is supported by NSFC (Grant numbers G0561671135, G0591630311, M0501020111531005). The authors thank the anonymous reviewers for their valuable comments.
Lipeng Xie: received the B.S. degree in communication engineering from Nanchang University, in 2013, and the M.S. degree from University of Electronic Science and Technology of China, Chengdu, China, in 2016. He is currently with the School of Communication and Information Engineering, University of Electronic Science and Technology of China as a Ph.D. student. His research interests include machine learning, image processing, medical image segmentation and object detection.
References (45)
- et al.
Assessment of algorithms for mitosis detection in breast cancer histopathology images
Med. Image Anal.
(2015) - et al.
Graph-based tools for microscopic cellular image segmentation
Pattern Recognit.
(2009) - et al.
Computer-aided diagnosis of breast cancer based on fine needle biopsy microscopic images
Comput. Biol. Med.
(2013) - et al.
Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases
J. Pathol. Inform.
(2016) - et al.
Morphological segmentation
J. Vis. Commun. Image Rep.
(1990) - et al.
A computer assisted method for leukocyte nucleus segmentation and recognition in blood smear images
J. Syst. Softw.
(2012) - et al.
DCAN: deep contour-aware networks for object instance segmentation from histology images
Med. Image Anal.
(2017) - et al.
Focal loss for dense object detection
IEEE Trans. Pattern Anal. Mach. Intell.
(2018) - et al.
Breast cancer histopathology image analysis: a review
IEEE Trans. Biomed. Eng.
(2014) - et al.
The changing histopathology of lung cancer. a review of 1682 cases
Cancer
(1977)
Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology
Proceedings of the 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro
Histological stains: a literature review and case study
Glob. J. Health Sci.
Histopathological image analysis: a review
IEEE Rev. Biomed. Eng.
Review of the current state of whole slide imaging in pathology
J. Pathol. Inf.
Computer-aided prognosis of neuroblastoma: classification of stromal development on whole-slide images
An automatic learning-based framework for robust nucleus segmentation
IEEE Trans. Med. Imaging
Robust segmentation of overlapping cells in histopathology specimens using parallel seed detection and repulsive level set
IEEE Trans. Biomed. Eng.
Improved automatic detection and segmentation of cell nuclei in histopathology images
IEEE Trans. Biomed. Eng.
Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images
IEEE Trans. Med. Imaging
Methods for nuclei detection, segmentation, and classification in digital histopathology: a review—current status and future potential
IEEE Rev. Biomed. Eng.
Nuclei segmentation in histopathology images using deep neural networks
Proceedings of the IEEE 14th International Symposium on Biomedical Imaging (ISBI)
ImageNet classification with deep convolutional neural networks
Cited by (54)
EMTL-Net: Boosting segmentation quality in histopathology images of gland and nuclei by explainable multitask learning network as an optimized strategy
2024, Engineering Science and Technology, an International JournalCNSeg: A dataset for cervical nuclear segmentation
2023, Computer Methods and Programs in BiomedicineOptimized detection and segmentation of nuclei in gastric cancer images using stain normalization and blurred artifact removal
2023, Pathology Research and PracticeAutomatic multi-tissue segmentation in pancreatic pathological images with selected multi-scale attention network
2022, Computers in Biology and MedicineA general deep learning framework for neuron instance segmentation based on Efficient UNet and morphological post-processing
2022, Computers in Biology and MedicineCitation Excerpt :Researchers mainly focus on improving the performance of CNNs, while the post-processing part is generally not explicitly described, although it is a critical step to obtain good segmentations. Applying watershed segmentation (WS) on the cell probability map derived from DL is the most common way [19,34]. Graph partition [35] and distance transform [36] are also popular techniques.
Multi-task generative adversarial learning for nuclei segmentation with dual attention and recurrent convolution
2022, Biomedical Signal Processing and ControlCitation Excerpt :In the meanwhile, an adversarial training method is adopted, and a new post-processing algorithm based on a voting mechanism is proposed to improve the confidence of patch border prediction in patch merging. The traditional nuclei segmentation algorithm mainly includes threshold-based method [11], watershed algorithm [12], and so on. However, these algorithms all require manual feature extraction, and the efficiency and accuracy of segmentation should be improved.
Lipeng Xie: received the B.S. degree in communication engineering from Nanchang University, in 2013, and the M.S. degree from University of Electronic Science and Technology of China, Chengdu, China, in 2016. He is currently with the School of Communication and Information Engineering, University of Electronic Science and Technology of China as a Ph.D. student. His research interests include machine learning, image processing, medical image segmentation and object detection.
Jin Qi: received the M.Sc. degree in mathematics from Sichuan Normal University, Chengdu, China, in 2002, and the Ph.D. degree in computer science from the Institute of Automation, Chinese Academy of Science, Beijing, China, in 2005. From 2005 to 2007, he was with the Department of Electrical Engineering, University of Electronic Science and Technology of China, as an Assistant Professor and from 2007 to 2009, as an Associate Professor. In 2010, he was a Visiting Scholar with the University of Chicago. From 2011 to Oct. 2013, he was a Postdoc with the Northwestern University. From Nov. 2013 to Oct. 2014, he was with the Medical College of Georgia. Since Nov. 2014, he has been with Childrens National Medical Center as a Postdoc. His research interests include fingerprint recognition, biometrics, image processing, computer vision, pattern recognition, and machine learning.
Lili Pan: received her B.Eng. degree in Electronic Engineering, as well as his M.Eng. and Ph.D. degrees in Information Engineering from School of Electronic Engineering at University of Electronic Science and Technology of China (UESTC), Chengdu, China in 2004, 2007, and 2012, respectively. From 2009 to 2011, she was a visiting student in the Robotics Institute at Carnegie Mellon University, Pittsburgh, USA. She joined Department of Information Engineering at UESTC as lecturer in 2012. She is now an associate professor of UESTC. Her research interests include computer vision and machine learning.
Samad Wali: received the B.Sc. degree in mathematics from Forman Christian College Lahore, Pakistan, the M.Sc. degree in applied mathematics from The Islamia University of Bahawalpur, Pakistan, and the Ph.D. degree in computational mathematics from Nankai University, Tianjin, China, in June 2018. His main areas of research interest are image processing, variational methods, and numerical solution to partial differential equations. He is currently a Research Fellow in Image segmentation and medical image analysis at the School of Communication and Information Engineering, University of Electronic Science and Technology of China.