Diabetic Retinopathy Grading by a Source-Free Transfer Learning Approach
Introduction
Diabetic retinopathy (DR) plays a major inducement of blindness in people aged 20–64 [1]. It is crucial for ophthalmologists to screen DR and track the developments in the early stage of disease, assisting patients against vision loss. Popular screening technology employs non-mydriatic colorful retinal cameras to collect fundus photos (Fig. 1), which expresses the essential characteristics to judge the DR grade. The most common lesion signs are often red or bright colors, indicating different lesion stages. The red dots existing in retinal images are mainly microaneurysms (MAs), focal dilatations of retinal capillaries. Besides, dot hemorrhage lesions (HEs) are found wherever capillary walls are weak inside the retina, slightly larger than MAs. Bright lesions or intra-retinal lipid exudates (EXs) result from the breakdown of the retinal blood barrier. Excluded fluid rich in lipids and proteins leave the parenchyma, leading to retinal edema and exudation. Moreover, progressive DR also causes macular edema, neovascularization (NV) and retinal detachment in later stages. In this situation, early detection of diabetic retinopathy guarantees patients have effective therapeutic effects. According to experts’ declaration, if the primary stage of DR can be detected, almost 90 percent of DR patients’ vision can be saved [2]. Ophthalmologists can manually complete DR screening based on the patient’s fundus images.
However, the symptom of early DR is tiny and the detection work in clinical practice is quite a lot. Meanwhile, the difference between two consecutive stages of symptoms is difficult to distinguish (Fig. 1). For instance, MAs are microscopic blood-filled bulges in the artery walls, which are the earliest signs of DR and are challenging to be noticed by ophthalmologists. HEs can be seen in moderate non-proliferative DR in addition to MAs, which are ‘blot’ shaped lesions slightly larger than MAs. This makes HEs can not be discriminated from MAs. Therefore, manual detection for many patients is inefficient and fatigable for ophthalmologists, which may produce misdiagnosis in long-time working. In recent years, many automatic DR detection methods are proposed with the development of artificial intelligence, which almost depends on deep learning technology. Therefore, the DR screening system developed through the artificial intelligence is much more authentic, reliable, faster, efficient, and easier than previous manual systems.
The current automated screening methods, especially using convolutional neural networks (CNNs), play a significant role in improving DR detection performance. For example, Abramoff et al. [3] proposed a deep-learning enhanced algorithm integrated with an IDx-DR device for the DR detection, which proves the effectiveness of the deep learning method compared with algorithms that only adopt machine learning. Shanthi el at. [4] used a modified AlexNet architecture to detect the DR stages and achieved better classification performance. Nevertheless, these methods are under a supervised framework, which requires a great deal of annotated data, causing several obstacles for clinical practice. Because marking sufficient retinal images is more time-consuming and economic-expensive than familiar images, it is very necessary to relax the limitation of the annotations. Here, transfer learning reveals its effectiveness for solving these problems due to its excellent capability of distilling knowledge from another dataset. Thus, it has received extensive research attention in recent years. For instance, Li et al. [5] firstly used a fine-tuned fully-trained CNN model to learn representations for inputting retinal images; Secondly, a robust support vector machine deployed these representations and suggested CNNs based domain adaptation methods have the capability to perform satisfactory grading accuracy on small datasets. Besides, Yang et al. [6] presented a residual-CycleGAN to distill the camera brand difference, which improves the transfer learning performance. These works support the potential of transfer learning and have reached a satisfactory performance in DR detection.
The remarkable performance of transfer learning methods in DR detection is reached by fully utilizing knowledge from adequate annotated data from the source domain [7], [5], [6], and these existing transfer learning models usually suppose that the labeled source retinal data is accessible during the training process. However, the huge number of annotated data may be not always available in the field of medical image analysis because of the following cases:
- 1.
Medical image annotation is a time-consuming and labor-intensive manual task. Besides, professional annotators need the friendly environment to keep excellent and accurate performance.
- 2.
Due to privacy and security issues of medical data, the source images are usually unavailable to access in transfer learning, whereas the sufficiently trained source model can be utilized in the training grading model for target data.
Therefore, it is necessary to develop an automated DR detection method without any annotated retinal images to accelerate the performance only with the help of a source model.
Motivated by the above observations, this study proposes a novel Source-Free Transfer Learning (SFTL) model for referable DR screening, which sufficiently exploits the learned source knowledge from the prediction model and the provided target images (Fig. 2). The proposed method mainly consists of a target generation module and a collaborative consistency module. Specifically, conventional data-based transfer learning methods [5], [6] aim to classify retinal images in the target domain with source images , labels and unannotated target images . In contrast, our proposed SFTL model is to distilling knowledge from the pre-trained classification model into the target data only giving . That is to say, learned model pre-trained by source data is accessible, but medical institutions do not provide source data .
To achieve this goal, we first introduce a generator and a discriminator for the target generation module to distinguish whether the input image is real or fake by an adversarial loss. Secondly, a target reconstruction loss is attached to the target generator to improve the generative performance. Besides, a semantic similarity constraint is imposed to collaborate with the classifier throughout the training process. In the collaborative consistency module, a model consistency loss acted on feature level and parameter level is designed to constrain the classifier from being too far away from the source model, making the training process staler and improving the prediction result. Moreover, the objective for the prediction model is integrated with a clustering-based regularization to make the features more compact in distribution space.
An extensive study on public retinal image datasets shows that our SFTL method can effectively solve retinal image classification by source-data-free transfer learning. The contributions of this paper can be summarized as below:
- •
We develop a Source-Free Transfer Learning (SFTL) model to transfer valuable information from a source pre-trained classification model, and further exploit predicted knowledge from unlabeled target domain using merely target retinal images, where none of the existing DR detection approaches is feasible.
- •
To enhance adversarial learning and improve the generative performance, we propose a target generation module attached a reconstruction loss, and a collaborative consistency module to avoid the features learned from the target prediction model drift far away from those by source pre-trained model, achieved by a feature consistency loss.
- •
We conduct transfer learning experiments between EyePACS and APTOS 2019 datasets, proving that our SFTL method can effectively improve the referable DR screening performance in the absence of source data.
Section snippets
Automated diabetic retinopathy grading
A series of research attempts have been devoted to improving the efficiency and accuracy of DR screening [8], [9], [10], [11], [12], [13], [14], [15]. We review the literature of DR screening by consolidating them into a bar graph based on the number of published papers in a year, as shown in Fig. 3. By using the technology of machine learning, deep learning, and transfer learning, remarkable progress has been made in automated DR screening.
Initially, researchers are inspired by machine
The proposed method
In this study, a novel transfer learning architecture is proposed to automatically grade retinal fundus image in the absence of source data, named the Source-Free Transfer Learning (SFTL) model. The overall architecture of SFTL model is shown in Fig. 4. In detail, our framework consists of two crucial modules of target generation and collaborative consistency, jointly trained from a pre-trained source classification model and massive unlabeled target images .
In the target generation module,
Experiments
In this section, sufficient experiments are performed on two datasets (EyePACS [37], and APTOS 2019 [38]) to prove that our proposed SFTL model can effectively execute the domain shift and increase the diabetic retinopathy diagnosis performance of the target prediction model. In our experiments, the EyePACS dataset and APTOS 2019 dataset are utilized as source and target domain, respectively.
We perform three steps of experiments: 1) Train a ResNet50 [19] model using source images as the source
Conclusion
This work exploits a novel transfer learning method without the help of source data for referable diabetic retinopathy diagnoses. Since many fundus images are hard to annotate and are not always accessible, our proposed SFTL method is more practical to train an effective DR diagnosing model. Both classification and generator modules have been reinforced reciprocally via collaborative optimization by incorporating generated target-style images into the transfer learning stage. Additionally, a
CRediT authorship contribution statement
Chenrui Zhang: Conceptualization, Methodology, Investigation, Writing – original draft, Formal analysis, Supervision. Tao Lei: Data curation, Validation, Writing – original draft, Resources. Ping Chen: Visualization, Investigation, Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This research was funded by the National Natural Science Foundation of China (61801437, 61871351, 61971381); Natural Science Foundation of Shanxi Province (201801D221206, 201801D221207); Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi (2020L0683); The National Natural Science Foundation of China under Grant 61461025, Grant 61871259, Grant 61811530325 (IECnNSFCn170396, Royal Society, U.K.), and Grant 61861024; The Key research and development plan of
References (50)
- et al.
Modified alexnet architecture for classification of diabetic retinopathy images
Comput. Electr. Eng.
(2019) - et al.
Identification and classification of microaneurysms for early detection of diabetic retinopathy
Pattern Recogn.
(2013) - et al.
Convolutional neural networks for diabetic retinopathy
Proc. Comput. Sci.
(2016) - et al.
Mustgan: Multi-stream generative adversarial networks for mr image synthesis
Med. Image Anal.
(2021) - et al.
Lgan: Lung segmentation in ct scans using generative adversarial network
Comput. Med. Imaging Graph.
(2021) - et al.
Super-resolution of cardiac mr cine imaging using conditional gans and unsupervised transfer learning
Med. Image Anal.
(2021) The use of the area under the roc curve in the evaluation of machine learning algorithms
Pattern Recogn.
(1997)- et al.
The evolving diabetes burden in the united states
Ann. Internal Med.
(2004) - et al.
Role of early screening for diabetic retinopathy in patients with diabetes mellitus: an overview
Indian J. Commun. Med.
(2011) - et al.
Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning
Invest. Ophthalmol. Visual Sci.
(2016)
Residual-cyclegan based camera adaptation for robust diabetic retinopathy screening
Application of higher order spectra for the identification of diabetes retinopathy stages
J. Med. Syst.
Dream: diabetic retinopathy analysis using machine learning
IEEE J. Biomed. Health Inf.
Classification of diabetic retinopathy images based on customised cnn architecture
Canet: Cross-disease attention network for joint diabetic retinopathy and diabetic macular edema grading
IEEE Trans. Med. Imag.
Classification of lesions in retinal fundus images for diabetic retinopathy using transfer learning
Imagenet classification with deep convolutional neural networks
Commun. ACM
Deep residual learning for image recognition
Cancer classification with data augmentation based on generative adversarial networks
Front. Comput. Sci.
Siamesegan: a generative model for denoising of spectral domain optical coherence tomography images
IEEE Trans. Med. Imaging
Cited by (32)
Source-free active domain adaptation for diabetic retinopathy grading based on ultra-wide-field fundus images
2024, Computers in Biology and MedicineA refined ResNet18 architecture with Swish activation function for Diabetic Retinopathy classification
2024, Biomedical Signal Processing and ControlJoint DR-DME grading classification using optimal feature selection-based deep graph correlation network
2023, Applied Soft ComputingTwo-stage framework for diabetic retinopathy diagnosis and disease stage screening with ensemble learning
2023, Expert Systems with ApplicationsAttention-Driven Cascaded Network for Diabetic Retinopathy Grading from Fundus Images
2023, Biomedical Signal Processing and ControlCitation Excerpt :Nowadays, early diagnosing retinopathy is considered as an effective way to stop sight degradation for blindness prevention [1–8].
HD<sup>2</sup>A-Net: A novel dual gated attention network using comprehensive hybrid dilated convolutions for medical image segmentation
2023, Computers in Biology and MedicineCitation Excerpt :However, the training of these early CNNs usually needed large amounts of training data. In view of this problem, many scholars adopted the transfer learning [4] in their training process [5–7]. In addition, the optimization of CNNs, including the training process [8] and the network structure [9], was also studied.