CS-CO: A Hybrid Self-Supervised Visual Representation Learning Method for H&E-stained Histopathological Images

Highlights

• Novel hybrid self-supervised visual representation learning method tailored for H&E-stained histopathological images.

• Generative and discriminative self-supervised learning can complement and enhance each other.

• Good rationality by leveraging domain-specific knowledge in histopathology.

• Good versatility for different kinds of computational histopathology tasks.

Abstract

Visual representation extraction is a fundamental problem in the field of computational histopathology. Considering the powerful representation capacity of deep learning and the scarcity of annotations, self-supervised learning has emerged as a promising approach to extract effective visual representations from unlabeled histopathological images. Although a few self-supervised learning methods have been specifically proposed for histopathological images, most of them suffer from certain defects that may hurt the versatility or representation capacity. In this work, we propose CS-CO, a hybrid self-supervised visual representation learning method tailored for H&E-stained histopathological images, which integrates advantages of both generative and discriminative approaches. The proposed method consists of two self-supervised learning stages: cross-stain prediction (CS) and contrastive learning (CO). In addition, a novel data augmentation approach named stain vector perturbation is specifically proposed to facilitate contrastive learning. Our CS-CO makes good use of domain-specific knowledge and requires no side information, which means good rationality and versatility. We evaluate and analyze the proposed CS-CO on three H&E-stained histopathological image datasets with downstream tasks of patch-level tissue classification and slide-level cancer prognosis and subtyping. Experimental results demonstrate the effectiveness and robustness of the proposed CS-CO on common computational histopathology tasks. Furthermore, we also conduct ablation studies and prove that cross-staining prediction and contrastive learning in our CS-CO can complement and enhance each other. Our code is made available at https://github.com/easonyang1996/CS-CO.

Introduction

Histopathology plays an important role in clinical medicine. It can reveal the morphology of pathologic cell and tissue at a microscopic level and provide vital information for disease diagnosis and prognosis (Srinidhi et al., 2021). In the past decades, thanks to the popularity of whole slide digital scanners, a growing number of histopathological slides have been digitized as histopathological images. This process of digitization not only facilitates the viewing, storing and sharing of histopathological slides but also paves the way for computer-aided analysis (Al-Janabi et al., 2012). In recent years, lots of computer-aided histopathological image analysis methods have been proposed, aiming to relieve the workload of pathologists and improve the objectivity of disease diagnosis (Gurcan et al., 2009). These valuable researches give birth to a promising research topic, computational histopathology, which has made a huge influence on the study of pathology (Abels et al., 2019). Furthermore, artificial intelligence-based computational histopathology has recently shown great promise to increase both the accuracy and availability of high-quality health care to patients (Srinidhi, Ciga, Martel, 2021, Cui, Zhang, 2021).

In computational histopathology, extracting effective visual representation is one of the most important problems (Gurcan et al., 2009). It is the cornerstone of many computational histopathology tasks, such as image retrieval (Shi, Sapkota, Xing, Liu, Cui, Yang, 2018, Yang, Zhai, Li, Lv, Wang, Zhu, Jiang, 2020), disease diagnosis (Shao, Bian, Chen, Wang, Zhang, Ji, et al., 2021, Lu, Williamson, Chen, Chen, Barbieri, Mahmood, 2021) and prognosis (Saillard, Schmauch, Laifa, Moarii, Toldo, Zaslavskiy, Pronier, Laurent, Amaddeo, Regnault, et al., 2020, Yao, Zhu, Jonnagaddala, Hawkins, Huang, 2020), and molecular signature prediction (Ding, Liu, Lee, Zhou, Lu, Zhang, 2020, Fu, Jung, Torne, Gonzalez, Vöhringer, Shmatko, Yates, Jimenez-Linan, Moore, Gerstung, 2020, Kather, Heij, Grabsch, Loeffler, Echle, Muti, Krause, Niehues, Sommer, Bankhead, et al., 2020). Besides, using visual representations instead of raw RGB images can significantly reduce data dimensionality and computational consumption. In earlier researches, some features are manually designed based on the knowledge of pathology and extracted via traditional feature extraction approaches. However, these traditional handcrafted features are very subjective, so their representation capacity is limited (Madabhushi and Lee, 2016). Recently, deep learning-based methods have shown powerful representation capability and have gradually become the mainstream for visual representation extraction (LeCun et al., 2015). Deep learning-based methods usually rely on large amounts of labeled data to learn good visual representations, while preparing large-scale labeled datasets is expensive and time-consuming, especially for histopathological image data. Therefore, to avoid this tedious data collection and annotation procedure, some researchers take a compromise and utilize pre-trained deep models, e.g. ImageNet (Deng et al., 2009) pre-trained convolutional neural network (CNN), to extract visual representations from histopathological images (Shao, Bian, Chen, Wang, Zhang, Ji, et al., 2021, Lu, Williamson, Chen, Chen, Barbieri, Mahmood, 2021, Saillard, Schmauch, Laifa, Moarii, Toldo, Zaslavskiy, Pronier, Laurent, Amaddeo, Regnault, et al., 2020, Yao, Zhu, Jonnagaddala, Hawkins, Huang, 2020, Ding, Liu, Lee, Zhou, Lu, Zhang, 2020). However, this compromise ignores both the data distribution difference and task bias, which will result in inappropriate or suboptimal visual representations.

Considering the aforementioned dilemma, self-supervised learning is one of the feasible solutions and has received increasing attention from researchers in recent years. The greatest advantage of self-supervised learning is that it can fit a deep model using only unlabeled data. Given a well-designed pretext task, the supervisory signals can be automatically generated from the unlabeled data. Then, the deep model can be easily trained to capture features by solving the pretext task in a supervised manner (Jing and Tian, 2020). In the past few years, self-supervised visual representation learning has made great progress. For natural images, several self-supervised learning methods (He, Fan, Wu, Xie, Girshick, 2020, Chen, Kornblith, Norouzi, Hinton, 2020) have achieved surprising results and shrunk the performance gap with supervised methods on downstream tasks (Jing and Tian, 2020). For histopathological images, a few self-supervised learning methods have also been proposed, but most of them have certain defects: some methods need side information other than images like magnification for supervision (Sahasrabudhe, Christodoulidis, Salgado, Michiels, Loi, André, Paragios, Vakalopoulou, 2020, Xie, Chen, Li, Shen, Ma, Zheng, 2020), others rely on the spatial proximity assumption which does not necessarily hold (Gildenblat, Klaiman, 2019, Abbet, Zlobec, Bozorgtabar, Thiran, 2020). As far as we know, there is still a lack of universal and effective self-supervised learning methods for extracting visual representations from histopathological images.

To this end, we propose CS-CO, a novel hybrid self-supervised visual representation learning method tailored for H&E-stained histopathological images. Our CS-CO employs two kinds of pretext tasks for self-supervised learning. One is the generative Cross-Stain prediction, and the other is the discriminative COntrastive learning. Both of them make good use of domain-specific knowledge and require no side information. Therefore, the proposed method has good rationality and versatility. The major contributions of our work are summarized as follows.

• We design a novel generative pretext task, i.e., cross-staining prediction, for self-supervised learning on H&E-stained histopathological images.

• We propose a new data augmentation approach, i.e. stain vector perturbation, to facilitate histopathological image contrastive learning.

• We integrate the advantages of generative and discriminative approaches and build a hybrid self-supervised visual representation learning framework for H&E-stained histopathological images.

• We demonstrate the superiority of the proposed CS-CO on several computational pathology tasks, such as patch-level tissue classification and slide-level disease cancer prognosis and subtyping.

This paper is an extension of the preliminary work (Yang et al., 2021) presented in MICCAI 2021. Besides a more in-depth background introduction, a more detailed method description and a more comprehensive discussion on experimental results, we improve and extend the previous paper in three main aspects. (1) We conduct new ablation studies to analyze the impact of the weighting of contrastive learning and cross-stain prediction on model performance. According to the experimental results, we refine the training strategy of CS-CO to improve the robustness. (2) We rerun the experiments presented in (Yang et al., 2021), with the more convincing cross-validation strategy and statistical significance testing. We also add two strong pathology-specific contrastive learning baselines for comprehensive comparison. (3) We evaluate the performance of CS-CO on two slide-level downstream tasks. One is for hepatocellular carcinoma (HCC) prognosis, and the other is for glioma subtyping. The experimental results indicate the effectiveness and versatility of CS-CO on common computational histopathology tasks.