Diabetic Retinopathy Grading by a Source-Free Transfer Learning Approach

Abstract

Diabetic retinopathy (DR) gives rise to blindness in young adults around the world. By early detection, patients with DR can be properly treated in time, and the deterioration of DR can be prevented. Thus, early and accurate DR screening is critical for disease prognosis. However, traditional manual detection work is intensive and easy to cause misdiagnosis for large amounts of patients. In recent years, deep learning methods have achieved remarkable improvements in medical image analysis, making DR detection more reliable and efficient. Nevertheless, existing supervised learning and transfer learning methods require a great deal of labeled data, which is always not available in DR screening due to the challenges of medical annotating and privacy issues. To solve this problem, we design a Source-Free Transfer Learning (SFTL) method for referable DR detection, which utilizes unannotated retinal images and only employs source model throughout the training process. In this paper, we propose two major modules, namely, the target generation module and the collaborative consistency module. For the target generation module, it can produce target-style retinal images, trained by the inputting target data and source model. For the collaborative consistency module, the classification model is further optimized by the generated target-style images, which guides the generator to produce images with more accurate expression. Furthermore, a target reconstruction loss is attached on the generator to enhance the performance, and a feature consistency loss is introduced to make the target model not drift far away from the source model. To evaluate the effectiveness of the SFTL model, we have carried out extensive experiments on APTOS 2019 dataset with a source model from EyePACS dataset, and obtained an accuracy of 91.2%, a sensitivity of 0.951 and a specificity of 0.858, demonstrating that our proposed SFTL model is more competitive than other state-of-the-art supervised learning methods.

Introduction

Diabetic retinopathy (DR) plays a major inducement of blindness in people aged 20–64 [1]. It is crucial for ophthalmologists to screen DR and track the developments in the early stage of disease, assisting patients against vision loss. Popular screening technology employs non-mydriatic colorful retinal cameras to collect fundus photos (Fig. 1), which expresses the essential characteristics to judge the DR grade. The most common lesion signs are often red or bright colors, indicating different lesion stages. The red dots existing in retinal images are mainly microaneurysms (MAs), focal dilatations of retinal capillaries. Besides, dot hemorrhage lesions (HEs) are found wherever capillary walls are weak inside the retina, slightly larger than MAs. Bright lesions or intra-retinal lipid exudates (EXs) result from the breakdown of the retinal blood barrier. Excluded fluid rich in lipids and proteins leave the parenchyma, leading to retinal edema and exudation. Moreover, progressive DR also causes macular edema, neovascularization (NV) and retinal detachment in later stages. In this situation, early detection of diabetic retinopathy guarantees patients have effective therapeutic effects. According to experts’ declaration, if the primary stage of DR can be detected, almost 90 percent of DR patients’ vision can be saved [2]. Ophthalmologists can manually complete DR screening based on the patient’s fundus images.

However, the symptom of early DR is tiny and the detection work in clinical practice is quite a lot. Meanwhile, the difference between two consecutive stages of symptoms is difficult to distinguish (Fig. 1). For instance, MAs are microscopic blood-filled bulges in the artery walls, which are the earliest signs of DR and are challenging to be noticed by ophthalmologists. HEs can be seen in moderate non-proliferative DR in addition to MAs, which are ‘blot’ shaped lesions slightly larger than MAs. This makes HEs can not be discriminated from MAs. Therefore, manual detection for many patients is inefficient and fatigable for ophthalmologists, which may produce misdiagnosis in long-time working. In recent years, many automatic DR detection methods are proposed with the development of artificial intelligence, which almost depends on deep learning technology. Therefore, the DR screening system developed through the artificial intelligence is much more authentic, reliable, faster, efficient, and easier than previous manual systems.

The current automated screening methods, especially using convolutional neural networks (CNNs), play a significant role in improving DR detection performance. For example, Abramoff et al. [3] proposed a deep-learning enhanced algorithm integrated with an IDx-DR device for the DR detection, which proves the effectiveness of the deep learning method compared with algorithms that only adopt machine learning. Shanthi el at. [4] used a modified AlexNet architecture to detect the DR stages and achieved better classification performance. Nevertheless, these methods are under a supervised framework, which requires a great deal of annotated data, causing several obstacles for clinical practice. Because marking sufficient retinal images is more time-consuming and economic-expensive than familiar images, it is very necessary to relax the limitation of the annotations. Here, transfer learning reveals its effectiveness for solving these problems due to its excellent capability of distilling knowledge from another dataset. Thus, it has received extensive research attention in recent years. For instance, Li et al. [5] firstly used a fine-tuned fully-trained CNN model to learn representations for inputting retinal images; Secondly, a robust support vector machine deployed these representations and suggested CNNs based domain adaptation methods have the capability to perform satisfactory grading accuracy on small datasets. Besides, Yang et al. [6] presented a residual-CycleGAN to distill the camera brand difference, which improves the transfer learning performance. These works support the potential of transfer learning and have reached a satisfactory performance in DR detection.

The remarkable performance of transfer learning methods in DR detection is reached by fully utilizing knowledge from adequate annotated data from the source domain [7], [5], [6], and these existing transfer learning models usually suppose that the labeled source retinal data is accessible during the training process. However, the huge number of annotated data may be not always available in the field of medical image analysis because of the following cases:

• Medical image annotation is a time-consuming and labor-intensive manual task. Besides, professional annotators need the friendly environment to keep excellent and accurate performance.

• Due to privacy and security issues of medical data, the source images are usually unavailable to access in transfer learning, whereas the sufficiently trained source model can be utilized in the training grading model for target data.

Therefore, it is necessary to develop an automated DR detection method without any annotated retinal images to accelerate the performance only with the help of a source model.

Motivated by the above observations, this study proposes a novel Source-Free Transfer Learning (SFTL) model for referable DR screening, which sufficiently exploits the learned source knowledge from the prediction model and the provided target images (Fig. 2). The proposed method mainly consists of a target generation module and a collaborative consistency module. Specifically, conventional data-based transfer learning methods [5], [6] aim to classify retinal images in the target domain with source images ${\mathbf{X}}_s$, labels $Y_s$ and unannotated target images ${\mathbf{X}}_t$. In contrast, our proposed SFTL model is to distilling knowledge from the pre-trained classification model $\mathcal{C}$ into the target data only giving ${\mathbf{X}}_t$. That is to say, learned model $\mathcal{C}$ pre-trained by source data is accessible, but medical institutions do not provide source data $\left\{{\mathbf{X}}_s,Y_s\right\}$.

To achieve this goal, we first introduce a generator and a discriminator for the target generation module to distinguish whether the input image is real or fake by an adversarial loss. Secondly, a target reconstruction loss is attached to the target generator to improve the generative performance. Besides, a semantic similarity constraint is imposed to collaborate with the classifier throughout the training process. In the collaborative consistency module, a model consistency loss acted on feature level and parameter level is designed to constrain the classifier from being too far away from the source model, making the training process staler and improving the prediction result. Moreover, the objective for the prediction model is integrated with a clustering-based regularization to make the features more compact in distribution space.

An extensive study on public retinal image datasets shows that our SFTL method can effectively solve retinal image classification by source-data-free transfer learning. The contributions of this paper can be summarized as below:

• We develop a Source-Free Transfer Learning (SFTL) model to transfer valuable information from a source pre-trained classification model, and further exploit predicted knowledge from unlabeled target domain using merely target retinal images, where none of the existing DR detection approaches is feasible.

• To enhance adversarial learning and improve the generative performance, we propose a target generation module attached a reconstruction loss, and a collaborative consistency module to avoid the features learned from the target prediction model drift far away from those by source pre-trained model, achieved by a feature consistency loss.

• We conduct transfer learning experiments between EyePACS and APTOS 2019 datasets, proving that our SFTL method can effectively improve the referable DR screening performance in the absence of source data.