A Novel Fundus Image Reading Tool for Efficient Generation of a Multi-dimensional Categorical Image Database for Machine Learning Algorithm Training

Machine learning has increasingly become the leading method of analyzing medical images.1 Fundus images have been the target of many studies, due to its relative simplicity and its significance in detecting leading causes of vision-threatening diseases, such as diabetic retinopathy (DR), age-related macular degeneration (AMD), and glaucoma. As these radiation-free images can be easily recorded by a fundus camera even without pupil dilation, they are suitable as a screening tool for preventing blindness. Deep learning algorithms have recently been reported to have high accuracy in detecting DR and other eye diseases.2, 3, 4, 5, 6 However, for effective deep learning, medical images that have accurate and detailed descriptions of their findings, diagnoses, and clinical significance are required, just as for the education and training of medical doctors. As archived medical images in healthcare provider's database generally have official readings and patient's diagnoses from clinicians, early investigators expected to easily use these information for machine learning-based research. However, these images with their clinic-based information fell short of their expectations. Archived clinic-based medical data are not robustly validated; most of them are read only once by a doctor, and a unified framework for readings barely exists. Also, the scope of readings varies depending on the purpose of the studies and clinically uninteresting findings are ignored which hampers the usage of data for training deep learning algorithms. Previous databases used for deep learning have focused on one specific diagnosis. As the interest in automated identification using deep learning has increased, there has been an increased demand on proper medical image data, and researchers have pointed out the lack of sufficient fundus image databases available for training deep learning models for efficient, high performing algorithms.7 Recently, Gulshan et al.2 suggested two prerequisites for high performing algorithms of deep learning; one is a large development set having large number of abnormal cases, and the other is a test/validation set having multiple readings per image.

To develop high performing algorithms for assessing retinal fundus images, a well-designed reading system for retinal fundus images is necessary to collect a large, comprehensive, reliable database which contains individual findings, diagnoses, and the clinical significance of the findings and diagnoses observed in the images from multiple graders. This retinal fundus image reading system can meet various purposes from ophthalmology clinics, primary care clinics, reading center for health screening center, and also for research-based image analysis. In the present study, we introduce our novel reading tool for retinal fundus images which produce multi-dimensional categorical information regarding image quality, presence of abnormality, findings, diagnoses, clinical significance and location information, which can provide a framework for belief network analysis as well as deep learning-based research. In addition, we report an overview of our database of multi-rated retinal fundus images using our reading tool by licensed ophthalmologists and the degree of agreement between the raters.

A total of 234,242 readings of 79,458 images were collected during the study period of about 6 months. Each rater read 278 to 15,499 images (mean = 4,280.0, median = 2,965) throughout all groups. Raters in group 2 read 23 to 14,998 images (mean = 4,632.2, median = 4,024), while those in group 3 read 10 to 8,078 images (mean = 2,743.4, median = 2,000), and those in group 4 read 218 to 4,077 images (mean = 2,467.4, median = 2,302).

Of 79,458 images, all 3 readings were completed in 77,007 images. Of these, images were excluded if even one reader graded it as unreadable in step 1. Thus, a total of 71,560 images, from 38,339 studies of 35,105 patients of readable image quality were included in the analyses, which is presented in Table 2. The number of images rated as abnormal in step 2 by at least one rater was 34,364, and out of these, at least two raters agreed in 24,024 images (69.9%) and all three raters agreed in 16,009 images (46.6%). In step 3, 378 (myelinated nerve fiber) to 13,781 (drusen/drusenoid deposit) images were rated to have a certain finding in them. Agreement of at least two raters on a certain finding ranged from 26.7% (retinal pigmentary change) to 65.2% (hemorrhage), and complete agreement of all three raters ranged from 5.7% (retinal pigmentary change) to 43.3% (hemorrhage). In step 4, the number of images with a certain diagnosis ranged from 185 (central retinal vein occlusion) to 10,611 (AMD), and agreement of at least two raters on a certain diagnosis ranged from 35.6% (other retinal/choroidal diseases/findings) to 65.6% (DR), and complete agreement of all three raters ranged from 11.0% (other retinal/choroidal diseases/findings) to 40.0% (DR). The number of images requiring referral to an ophthalmologist rated by at least one rater in step 5 was 17,798, agreement of at least two raters on requirement of referral was 27.8%, and complete agreement of all three raters was 5.1%.

Table 2
An overview of the reading results using the retinal fundus image rating system

• Click for larger image
• Click for full table
• Download as Excel file

When analyses were confined to the 16,009 images that had complete agreement for abnormality in step 2 (Table 3), complete agreement rate improved in all variables, especially for the following variables; drusen/drusenoid deposit, retinal nerve fiber layer defects (RNFLDs), and glaucomatous disc change in step 3, glaucoma suspect in step 4, and requirement of referral in step 5.

Table 3
Complete agreement rate (all 3 raters agreed) in images with complete agreement on abnormality

• Click for larger image
• Click for full table
• Download as Excel file

In the analysis of complete agreement rate in specific groups (Table 4), retinal specialists showed superior agreement regarding retinal findings and diagnoses than those observed among general ophthalmologists and glaucoma specialists. Complete agreement for any AMD was 31.4% in group 3 versus 16.6% in group 2 and 22.8% in group 4, and complete agreement for any DR was 55.6% in group 3 versus 31.6% in group 2 and 45.6% in group 4. Similarly, glaucoma specialists' agreement in glaucoma-related findings (RNFLD and glaucomatous disc change) and diagnosis (glaucoma suspect) was remarkable, compared to general ophthalmologists and retinal specialists. Complete agreement for glaucoma suspect was 38.5% in group 4 versus 5.5% in group 2 and 9.4% in group 3.

Table 4
Complete agreement rate (all 3 raters agreed) according to rater's subspecialty group

• Click for larger image
• Click for full table
• Download as Excel file

We developed a novel comprehensive reading system for retinal fundus images, and using this system, we collected a valuable dataset of over 200,000 multi-dimensional readings for about 80,000 images with three readings per image by licensed ophthalmologists in Korea in a relatively short period of only about 6 months. This on-going database currently contains 34,364 images with at least one rating as abnormal, and 16,009 images with complete agreement as abnormal, and in each abnormal image, detailed findings are annotated along with location information of the findings, supposed diagnoses, and clinical significance of the diagnoses. This dataset can be used in various ways according to the purpose of the researcher. The present database can provide a sufficient number of images even with complete agreement for each finding or diagnosis. Further on, more images can be included if the level of agreement is decreased to 1 or 2, depending on the purpose. We can extract specific images with certain findings in step 3 with a certain location irrespective of the diagnosis in step 4 or can extract images with a specific diagnosis in step 4 with various findings in step 3. Using the information acquired from step 4 from our reading system, we can develop an algorithm diagnosing or classifying a specific retinal disease, such as DR as in previous studies,2, 3, 5 Further on, this database can be used to train or test algorithms that can detect the presence of specific findings in a certain diagnosis (i.e., hemorrhage, cotton wool patch, hard exudate, membrane, etc. in DR). Therefore, the data from our reading system can support both strategies in developing an algorithm for assessing retinal fundus images; one, assessing findings from the images and then drawing the diagnoses from probabilistic relationship between findings, diagnoses, and their clinical significance, and two, directly determining the diagnoses from images. We are currently undergoing development of deep learning algorithms utilizing both of these approaches with our database created from this study. Most developers have focused on developing models with a diagnosis at the image level. However, in clinical practice, there is a greater diversity; one fundus image may have several simultaneous diseases, and a certain disease may present with diverse findings. There have been approaches to develop devices that use deep-learning-based finding detectors.4 Also, Quellec et al.8 have proposed that deep learning models trained to detect a certain diagnosis can also be excellent lesion detectors without retraining them, by creating heatmaps showing which pixels play a role in the image-level predictions. Developing image databases that can be used in both of these two approaches is increasingly becoming important, and this novel reading system supports the creation of such a database with multi-level annotation in a relatively short time.

One problem in the application of deep-learning-based diagnosis devices in real clinical practice is that there may be a diverse range of findings that could be found in the retina. Previous binary classification models for DR have reported high accuracy in automated DR detection, however, the limitation of clinical application of this device in the real world has been acknowledged,9 and development of deep learning algorithms categorizing more than one retinal disease have been reported.5, 7, 10 Another challenge, as Choi et al.7 reported, is that as the number of disease categories are increased, the performance of deep learning models are diminished. The dilemma is that the real world is far more diverse and systems with training on only a few disease categories are more likely to face unlearned disease. Thus, systems with multi-categorical classification should be developed, but the performance decreases with more classification. A practical meeting point in between needs to be found. Our system was developed to categorize findings and diagnoses most commonly encountered in subjects visiting primary care clinics and health screening centers. We excluded some vision-threatening and/or rare findings/diagnoses such as retinal detachment and retinal artery occlusion, which may cause acute and severe deterioration of vision with rather easily recognizable symptoms, which would bring the patient directly to the clinic rather than be discovered in a screening setting for normal subjects. The simplicity of the findings and diagnoses in our system may improve the efficiency of developing a proper algorithm for screening purposes. In addition, simplifying the findings (e.g., various types of hemorrhage [preretinal, retinal, subretinal, disc hemorrhage, etc.] were simplified to one category of hemorrhage in the present system) may also help reducing the number of categories.

Unlike other public datasets including EyePACS and Messidor11, 12 or other in-house medical data2, 5 that yield only single annotation per image, our system can produce data that are multi-annotated per image. In other words, in a single image, precise and thorough clinical information can be recorded which in turn can also be used for multi-faceted analyses and for designing semantically rich machine learning models. Furthermore, binding the locational information with findings is one of the key motivations in developing our novel reading system. Moreover, rather than using a simple grid format for location, our geometric regions are designed to straightforwardly reflect anatomical characteristics of the human eye and to practically capture clinically frequent findings. Not only is it convenient for doctors to communicate with each other with the more physiologic region information, but also the practical region division enables an efficient way for describing the state of the findings which can help ease the development of a machine learning system that can identify not only which findings are present but also where the findings are located.

In addition, as we have divided the system in Groups according to the raters' subspecialty (i.e., general ophthalmologists, retina specialists, and glaucoma specialists), which significantly positively affected the reliability of the reading for the corresponding subspecialty. As expected, retina subspecialists showed higher agreements in retinal findings and diagnoses than other raters, while glaucoma specialists did better in glaucoma findings and diagnoses than other raters. Agreement may be higher in the setting of clinical practice with dedicated healthcare providers, with other clinical background information also available, than in the research setting of this present study. Differences in career duration and experience, affiliated institute, and subspecialty between raters may have affected the annotation of ambiguous findings (e.g., asymptomatic aging changes with less clinical significance), which may also have led to disagreements. When the analyses were confined to images that had complete agreement for abnormality in step 2, the agreements regarding findings and diagnoses improved significantly, especially in drusen/drusenoid deposit, RNFLD, and glaucomatous disc change. This is possibly due to the exclusion of images with these borderline ambiguous findings. The quality and the level of truth of the readings of images used in machine learning is considered to have increasing importance in recent studies. Recently, a small set of adjudicated DR grades by retinal specialists showed substantial improvements in algorithm performance.13 Datasets with more readings per image with higher ground truth have been reported to show higher performance.2 The image set of our database can be grouped according to the raters' subspecialty, and the level of confidence may be increased to use in settings requiring higher level of evidence.

Even with these advantages, our reading system for retinal fundus images also has several limitations. Various factors may have affected the rather low inter-rater agreement in our study. The raters in the present study were asked to read images only for research purpose without any other demographic and clinical information except age and gender. The non-clinical use and lack of other clinical and ophthalmologic information may have affected the readings and their agreements. Providing the raters with other clinical information such as the presence of systemic conditions including hypertension and diabetes mellitus, ophthalmologic history and ophthalmologic measurements such as visual acuity or intraocular pressure, status of the contralateral eye, and previous examination images may enhance the accuracy of each individual raters' decisions. The individual grading environment of raters was not standardized, and inter-rater agreement may have been sacrificed with relatively poor quality of the display system in some raters. Detailed and precise criteria for certain ambiguous findings may have also leaded to improved agreement among readers. Another limitation of our system is that we simplified the number of findings and diagnoses available, as the system was first intended to be developed for the purpose of screening rather healthy individuals. The present system grouped these other findings/diagnoses as a separate grouped variable, therefore we could re-read these images using a modified reading system if needed in the future. The location information of the findings was not at the pixel level, but in compartments defined by normal anatomical structures in the fundus, which may limit use in future studies requiring pixel level location information. Our system was designed to grade only posterior pole images without information on the area outside the angle of the fundus image. Image quality such as media opacity, small pupil, defocusing was not put into account in this analysis, but information on image quality was also collected, and may be considered in future analyses using this database.

In conclusion, our reading system can provide efficient generation of retinal fundus images with detailed, high-quality multi-dimensional categorical information including image quality, findings, diagnoses, and clinical significance, which provides a framework for understanding how ophthalmologists assess retinal fundus images according to 5 each rating step and can be effectively utilized in developing algorithms for assessing and visualizing findings as well as for determining diagnoses from retinal fundus images. The degenerated data present various degrees of agreements between graders according to findings and diagnoses, which emphasize the importance of addressing the grader variability in the development of algorithms.