Accuracy of Low-Cost, Smartphone-Based Retinal Photography for Diabetic Retinopathy Screening: A Systematic Review

Introduction

Diabetic retinopathy (DR) is a common diabetic retinal microvascular complication found in approximately one in every three individuals with diabetes,^1,2 which potentially leads to irreversible blindness if left untreated. Evidence has suggested that 3.7 million people are blind or visually impaired due to DR. More importantly, this number has increased nearly 1.5-fold from 1990 to 2010.³

Routine screening and timely treatment are key for successful management of DR to avoid visual loss.^4,5 Once identified early, timely treatment for DR may reduce the risk of DR progression and visual loss by 50% in a year,⁶ emphasizing the importance of early DR screening in diabetic patients despite any symptoms. Both the International Council of Ophthalmology (ICO) and the American Diabetes Association (ADA) recommended at least annual screening of visual acuity and retinal examination for every person with diabetes to avoid delayed treatment.^7,8 Retinal photography for diagnostic purposes of DR can be done either as a stereoscopic or two-dimensional photograph, with a ≥30° field, with or without mydriatic drugs, with or without optical coherence topography (OCT). However, retinal photography for community DR screening is still not adequately available in many countries, particularly in countries with lower health financial resources.^9,10

In the last two decades, rapid advancement of digital fundus photography has resulted not only in picture quality but also increased portability of the devices.^11,12 At the same time, there is also advancement of a smartphone camera system that aligns with this, allowing researchers and industries to create smaller and more affordable devices fitted to a smartphone camera system that captures retinal images.^13,14 While the quality may not be as good as a standard, table-top fundus camera, these systems may improve the cost-effectiveness of DR screening and will potentially change the outlook of DR screening strategies in low resource settings in the near future.

Recent reviews and meta-analyses have provided extensive discussion and comparisons of various imaging modalities in DR grading, including portable, smartphone-based fundus imaging in DR grading, and some have also reported their sensitivity and specificity values.^15–17 However, none of these articles specifically focused on low-cost devices. In this paper, we will systematically review all available low-cost, smartphone-based fundus camera systems and reported their accuracy for DR screening in patients with diabetes. This review will complement previous reviews and provide additional evidence and understanding about the potential role and importance of low-cost, smartphone-based devices for DR screening, particularly in low resource settings.

Materials and Methods

Search Strategies

The protocol for this review was registered in PROSPERO in June 2021 under the registration number CRD42021249746.¹⁸ We implemented our search strategy that has been developed for MEDLINE through PubMed and adapted it to search literature from other electronic databases and registers including Scopus, Google Scholar, and Cochrane Central Register of Controlled Trials (CENTRAL). We used the following combinations of keywords: “diabetes mellitus”, “smartphone fundus photograph”, and “diabetic retinopathy”. Only studies published in English and studies of human subjects were filtered. There were no restrictions for date of publication. We also manually searched from the reference list of all primary studies to see if there were any relevant studies to be included. Further details regarding our keywords and search strategy are included in Supplementary Table 1–4.

Study Selection

We included studies involving smartphone-based device(s) that: 1) assessed diagnostic test accuracy of the device for detecting DR in diabetic adult older than 18 years; 2) compared one or more devices with a reference standard that has been widely accepted to diagnose DR such as: all types fundus photography, slit-lamp bio-microscopy, direct ophthalmoscopy, indirect ophthalmoscopy; 3) using low-cost smartphone-based devices with either a direct or indirect ophthalmoscopy concept that has a retail price range less than $700 (approximately equivalent to IDR 10,000,000); and 4) reported sensitivity and specificity or had sufficient data to develop a 2×2 table. There were no limitations regarding whether health professionals or trained examiners operated the smartphone-based device, the use of mydriatic drugs, the grading process, the materials of the device, and the smartphone details (brand, series, manufacturers). Studies were excluded if they used sophisticated LED illumination externally attached to the smartphone.

Title and abstracts of studies that met the inclusion criteria were reviewed by four personnel (MEP, MBS, AFZ, or RMI) independently. Disagreements were resolved through discussion by MEP, MBS, and AFZ, followed by selecting the included full texts and consolidation of disagreements. We extracted the following data from included studies: 1) author; 2) year of publication; 3) participant characteristics (eg, sample size, country setting, mean age, mean duration of diabetes); 4) index test characteristics (eg, type of compatible smartphone, illumination source, type of lens, ophthalmoscopy method); 5) imaging protocol for index test (eg, the use of mydriatic drugs, retinal field of view, working distance, image format, and resolution); 6) reference test; and 7) test outcomes of sensitivity and specificity. We further contacted the corresponding author to request additional data (ie, sensitivity and specificity) when these were not available from the text. We also performed searches in various marketplaces to obtain the average price for each device. An Excel database was created to facilitate our reviewer in selecting eligible studies and recording extracted data.

Risk of Bias and Analysis

We used the QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies) tool to assess the risk of bias and applicability of all included studies. Two reviewers performed the assessment independently (MEP and AFZ) and any disagreements were resolved through discussion with all the authors.

The unit of assessment we used to calculate the sensitivity and specificity of a smartphone-based device was the proportion of participants. For the purpose of our analysis, we categorised the outcomes into three groups to analyse the accuracy of smartphone-based devices to diagnose any DR, diabetic macular oedema (DMO), and referrable DR (moderate non-proliferative DR [NPDR] with DMO or severe NPDR or worse, with or without DMO) based on clinical grading using retinal photographs without any OCT examination.⁷ Studies that did not include sensitivity and specificity of these groups of interest but reported other detailed test outcomes such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN) numbers were included and analysed using a 2×2 table to acquire the sensitivity and specificity values. These data were presented in a forest plot created using RevMan 5.4.

Results

Search Results

Our keywords search strategy resulted in 294 records. After title and abstract screening, 17 articles were considered relevant for the full text retrieval. One study article was a conference abstract, thus only 16 full text articles were further assessed. From full text screening, one study used synthetic eyes as test subjects, nine were embedded with high-cost/sophisticated devices and assessed other interventions’ accuracy (AI, tablet, camera, portable fundus camera device called EyeScan and high-cost smartphone-based fundus camera called Remidio Fundus on Phone [FOP]), and two studies did not include any accuracy values, leaving only five studies that met our inclusion criteria. This selection process is detailed in Figure 1.

Figure 1 Prisma flow diagram regarding study selection process. Note: *Other interventions including artificial intelligence (3 studies), tablet (1 study), DSLR camera (1 study), hand-held fundus camera (1 study), and high-cost smartphone-based device (3 studies).

Characteristics of Included Studies and Assessment of Biases

Studies included in this review were conducted in the United States of America (USA),^14,19 Cameroon,¹³ Italy,²⁰ and India,²¹ with a sample size ranging from 50 to 220 participants. Participants’ mean age ranged from 56.7 to 60.5 years and diabetes duration ranged from 7.0 to 11.9 years. Four studies used the International Classification of DR (ICDR) whereas only one used the Modified Early Treatment of Diabetic Retinopathy Study (ETDRS) grading system (Table 1).

Table 1 Characteristics of Included Studies

Figure 2 illustrates the risk of bias assessment of included studies. All included studies had potential biases from an unclear patient selection due to inadequate reporting of the recruitment strategy (ie, failure to report exclusion criteria or sampling method). There was one study by Kim et al¹⁴ that had potential biases from insufficient documentation of patient selection, index test, reference standard, and flow and timing of the examinations. The number of participants included in the analysis were not reported in two studies.^14,21

Figure 2 Risk of bias assessment using QUADAS 2 tools.

Diagnostic Accuracy for DR, Referrable DR, and DMO

Detailed characteristics and image protocol of the smartphone-based devices reported in this review are presented in Table 2. There were six different smartphone-based devices reported in included studies (Peek Retina,²¹ D-EYE,^20,21 Do-it-yourself solution by Sankara,²¹ Paxos Scope,^19,21 MII RetCam,¹³ and CellScope Retina¹⁴). All devices were compared with indirect ophthalmoscopy examination as the reference standard, except for the D-EYE²⁰ that also used slit-lamp examination and CellScope Retina¹⁴ that only used slit-lamp examination for the reference. The principal work of three devices (Peek Retina, D-EYE, and DIY) were similar to direct ophthalmoscopy while the others (Paxos Scope, MII Ret Cam, and CellScope Retina) were similar to indirect ophthalmoscopy. All devices required fully dilated pupils for the examinations. Some of the studies did not include the observed field of view, so the review authors made assumptions based on the retinal photographs taken by those devices that were included in the studies.

Table 2 Smartphone-Based Devices’ Accuracy, Characteristics, Image Protocol, and Retail Price Range

The diagnostic accuracy of the six devices is summarized in Table 2. We found that the sensitivity for detecting any DR, referrable DR, and DMO ranged between 52–92.2%; 21–91.4%; and 29.4–81%, respectively, with the highest sensitivity acquired using the CellScope Retina, except for detecting DMO (the highest sensitivity was achieved using the D-EYE). Meanwhile, the specificity for detecting any DR, referrable DR and DMO ranged between 73.3–99%; 64.9–100%; and 95–100%, respectively. The accuracy in diagnosing referrable DR was not assessed for the MII Ret Cam device.

Four of five articles included in this study presented their original data following five DR severity levels: no DR, mild NPDR, moderate NPDR, severe NPDR, and PDR, and also presented data for DMO, with the exception of the study by Toy et al.¹⁹ Two of these articles further re-categorized these severity levels into simplified DR classification for clinical purpose: any DR or non-referrable DR, and referrable DR.^14,19 Calculated sensitivity and specificity values of these devices were based on this classification. Wintergerst et al²¹ is the only exception, as they only presented concise classification of any DR, referrable DR, and DMO.

Mii Ret Cam showed wide variation of sensitivities in diagnosing five DR severity level. The lowest sensitivity was reported in the diagnosis of moderate NPDR (43%) and the highest in PDR (100%).¹³ D-EYE sensitivities also varied between 55–96%, but the lowest sensitivity was for diagnosing severe NPDR and the highest was for normal fundus without DR. On the other hand, sensitivities for diagnosing moderate NPDR and PDR were both good, which were 82% and 89%, respectively.²⁰ Similar results were demonstrated by CellScope Retina, in which the lowest sensitivity was for diagnosing mild NPDR (33%) and the highest was for diagnosing any DR (94%) and PDR (72%).¹⁴ Paxos Scope has the lowest sensitivity among other devices in diagnosing mild NPDR (0%) and the highest sensitivity for severe NPDR and PDR (100%).¹⁹

Specificities for diagnosing DR in all four articles were more comparable, except for CellScope Retina. Paxos Scope has a specificity ranging between 99–100%, with the highest for severe NPDR and PDR.¹⁹ Both Mii Ret Cam and D-Eye had their lowest specificity for mild NPDR (90% and 93%, respectively) and the highest for PDR (both 100%).^13,20 CellScope Retina showed substantial variation in specificity: 40% for any DR and 94% for PDR.¹⁴

Four out of six devices had retail price ranges less than $700, with the exception of DIY by Sankara, which is a modification to the smartphone that can be assembled by ourselves, and CellScope Retina, which is not commercially available at this time.

Meta-Analyses

Any DR

Four out of five studies (679 participants) presented data for any DR^13,14,19,20 and were evaluated using a Forest plot, as shown in Figure 3. The sensitivity ranged from 72–94%, and specificity from 40–99%. The most extreme values from studies with smaller sample sizes and wide 95% CIs was found in CellScope Retina.¹⁴ One study did not contribute data to the meta-analysis because sensitivity was not estimable.²¹

Figure 3 Forest plot for sensitivity and specificity of any DR, referral DR, and DMO.

Referral DR

Three out of five studies presented data for referral DR from a total of 459 participants.^14,19,20 Sensitivities for referral DR ranged from 88–93%, and specificities from 57–98% from a total 459 participants of the three studies (Figure 3).

DMO

In DMO groups, three out of five studies with a total of 573 participants^13,14,20 contributed to the estimation of sensitivity, which ranged between 47% and 81%. The most extreme values were from CellScope Retina which had smaller sample sizes and wide 95% CIs.¹⁴ Specificity was more homogenous, ranging from 91–100% (Figure 3).

Discussion

In this systematic review, we reported the sensitivity and specificity of six low-cost, smartphone-based devices in detecting DR, referrable DR, and DMO when compared with indirect ophthalmoscopy and slit-lamp bio-microscopy as the reference standard. These devices showed considerably good sensitivity (52–92%) and specificity (73–99%) when detecting the presence of any DR. However, a wider range of sensitivity was reported when detecting referrable DR (21–91%) and DMO (29–81%), as opposed to their high specificity for referrable DR (65–100%) and DMO (95–100%). This suggests that the use of low-cost, smartphone-based devices for DR screening in the community should be acceptable, particularly for countries or areas with a lack of facilities, difficult geographical features, or health financing constraints.

There were a very limited number of previously published studies for comparison. A recent systematic review and meta-analysis by Tan et al¹⁷ also analysed six different smartphone-based devices from nine articles, but only three of those devices met our inclusion criteria. Three articles featured a high-cost device (more than $700) called Remidio FOP; one of them featured a handheld 20D lens that does not use an adaptor; and one article featured a device called Ocular CellScope that was an early design of CellScope Retina.¹⁴

The accuracy of these low-cost devices were nearly comparable to the recommended retinal examination for DR screening using either direct/indirect ophthalmoscopy, slit-lamp examination, or traditional table-top fundus cameras with a ≥30° field of view.⁷ The study by Baeza et al²² compared dilated 45° single-field photographs using a table-top fundus camera with gold standard of seven standard stereoscopic 30° field photographs as proposed by ETDRS²³ and found sensitivity and specificity for detecting any DR of 77% and 98%, respectively, and for detecting referrable DR of 82% and 99%, respectively. A similar study by Murgatroyd et al²⁴ using a slit-lamp examination as reference standard shows sensitivity and specificity for detecting any DR of 86% and 91%, respectively, and for detecting referrable DR of 81% and 92%, respectively. A Veteran Affairs Diabetes Trial (VADT) study compared clinical examination using both direct and indirect ophthalmoscopy with standard 7-field ETDRS fundus photographs and found that the sensitivity and specificity for detecting any DR were 51% and 91%, respectively. In addition to any DR, the sensitivity and specificity for detecting PDR were 61% and 98%, respectively. However, the sensitivity for detecting DMO was low (24%), in contrast to its high specificity (98%).²⁵

Rapid technological advances in retinal imaging have improved the accuracy and time consumption of DR detection.²⁶ Moreover, with the presence of artificial intelligence (AI), efforts or personnel needs to perform the screening have reduced. Landmarks studies have documented that machine learning system could detect referrable DR from retinal photographs with sensitivity and specificity of more than 90% when compared with expert decision.^27,28 For example, one of the latest technologies in AI for DR detection was the development of an active deep learning (ADL) method using an artificial bee colony (ABC) algorithm. This method has been shown to have enhanced ability to detect five levels of DR severity whilst an earlier AI method was only able to detect two DR levels: referrable and non-referrable DR.²⁹ These results indicated its potential in clinical scenario to enhance efficiency in DR screening coverage. However, much attention has mostly focused on the development of a DR screening system which involved the use of sophisticated or expensive equipment that are less portable and less affordable for low resource countries.

In this review, we found only five out of hundreds available studies in the literature have attempted to use low-cost devices to screen DR. Unlike currently established DR screening systems that use a table-top fundus camera, low-cost smartphone devices have not gained much interest possibly due to a priori assumption that the accuracy would be low.^13,21 This review clearly suggested otherwise, that accuracy of the available low-cost devices was not far behind the table-top camera system and was suitable in the context of DR screening.³⁰ Relatively lower sensitivity may indicate that these smartphone-based devices may have missed included early or no DR cases into referrable DR group or recognized cases without DMO as having DMO. However, high specificity in detecting referrable DR and DMO has strongly emphasized that once a case is not detected as referrable DR or DMO, that case is less likely to have referrable DR or DMO needing further treatment.

It is noteworthy that small field of view could be one prominent limitation of smartphone-based devices when compared with table-top fundus camera. Four out of five devices we included in this review have a field of view ranging from 20–40°, barely in accordance to recommended ICO screening guidelines.⁷ Prior review had shown that most studies regarding DR screening using a table-top fundus camera capture at least a single 45° field of view and others had claimed the importance of a wider field of view to identify DR characteristics that may occur in the peripheral retina.^30,31 A single 60° field of view was found to improve the screening process because it can still detect microaneurysm lesions and referral DR with a lower number of capture. Remidio Vistaro was reported to provide a 65° field of view and the montage of two fundus photographs could exceed the standard 7-field ETDRS view.^32,33 The latest technology of ultra-wide field (UWF) imaging with a ≥100° field of view was found to be very effective in detecting peripheral DR lesions because it can capture around 82% of the retinal surface.³⁴ A study compared UWF retinal imaging with a 200° field of view and standard 7-field ETDRS and found 51% of the DR lesion was found within the standard ETDRS view, 15% found in the peripheral outside the standard ETDRS view, and 34% were distributed evenly.³⁵ In order to overcome this problem, there are alternative imaging protocols than can be done to capture a wider field of view beyond the initial capabilities of these devices, such as using a montage of several photographs (eg, CellScope Retina) or using video mode (eg, Peek Retina, D-EYE, Paxos Scope).

There are some important implications of this review. To date, nationwide systematic DR screening has been fully implemented only in very few countries, such as the UK and Ireland.^31,36 Other high resource countries, for example the USA, Singapore, and European countries, are progressing substantially but have not yet established the same system as the UK.⁹ On the other hand, developing countries or countries with low health resources are mostly struggling with the provision of equally accessible screening and treatment facilities in each area in the country due to financial barriers.^9,37 In the context of Indonesia as an example, Indonesia is one of the developing countries having a growing burden of diabetes but multiple problems related to DR screening: 1) difficult geographical features; 2) inadequate health infrastructures and access in rural areas; 3) uneven distribution of eyecare personnel; and 4) a low government budget for eye healthcare. Devices included in this study have a market price below $700 (ranged between $134–$435), which are more affordable compared to a standard table-top fundus camera such as Zeiss Visucam Pro NM that has the average market price range of $10,000. This review may propound that low-cost, smartphone-based devices can significantly reduce the financial burden of DR screening which is heavily related to providing a large amount of fundus cameras and frequent retinal imaging.³⁸ Several devices we included in this study had been trialled in developing countries that have similar problems with ours: Peek Retina was used for screening in Uganda and Mii Ret Cam in India.^13,39 More importantly, these devices are simple and should not be difficult to manufacture. Therefore, with the current state of advancement in smartphone technology, researchers and industries working in this area should be more encouraged to develop similar devices.

There were several studies which were excluded because it features smartphone-based devices that has retail prices over $700. Remidio FOP was one of the excluded devices, which is a product manufactured in India that employs an indirect ophthalmoscopy method, can be used without mydriatic, has a 45° field of view and lens adjustment between −20 D to +20 D.⁴⁰ This device had gone through clinical validation to diagnose DR and, compared to standard 7-field fundus photography, it has a high sensitivity and specificity for any DR (93% and 98%, respectively), referral DR (88% and 95%, respectively), and DMO (87% and 95%, respectively).⁴¹ Unfortunately, we found that the retail price for Remidio FOP far exceeds our definition of low cost, which is between $5,000-$8,000. Another device we excluded was Eyer, which was a smartphone-based device produced in the United States. Similar to Remidio FOP, Eyer also employs an indirect ophthalmoscopy method, hasa 45° field of view and autofocus range from −20 to +20 D. This device also went through clinical validation and has sensitivity of 91% and specificity of 81% when compared to single-field table-top fundus photography for referrable DR. It also showed a relatively good agreement (73%) for diagnosing all six levels of DR severity. However, the retail price also exceeded our definition of low cost, which is around $4,500.⁴² Out of the six smartphone-based devices we included in this review, Peek Retina is the only device that had gone through clinical validation to diagnose DR. Compared to a standard ophthalmic fundus camera, it has good sensitivity and specificity for diagnosing any DR (84% and 79.9%, respectively).³⁹ Peek Retina also had validation studies for optic disc imaging, which shows excellent agreement (kappa coefficient of 0.69) between the smartphone-based device and the standard ophthalmic fundus camera.⁴³ These findings are similar to another validation study done in Brazil to evaluate smartphone-based devices in measuring cup-to-disc ratio, which also shows excellent agreement.⁴⁴

The strength of our study is the use of a detailed search strategy pre-defined in our study protocol. However, limitations are noted. First, because not all studies presented detailed data needed for our calculations, we were only able to perform simple meta-analyses using a Forest plot. Second, we did not redefine the definition of referrable DR used in included studies. However, there were only slightly different definitions of referral-warranted DR which all were referred to definitions used in prominent studies.⁷ Finally, there were also potential biases associated with the reference standards being clinical examinations instead of retinal photography using a table-top fundus camera and the process of capturing retinal images that were influenced by image qualities and smartphone camera specifications.

Conclusion

In conclusion, this review found that currently available low-cost, smartphone-based devices showed a relatively wide range of overall sensitivity but more consistent specificity for detecting any DR, referrable DR, and DMO. The accuracy of these devices was not far behind the high-cost DR screening systems, suggesting the potential use of low-cost, smartphone-based devices for DR screening in countries that struggle to provide a high-cost DR screening system. More importantly, this review may enlighten researchers in this area that there are opportunities to develop more affordable smartphone-based devices with better accuracy to increase the availability of affordable DR screening equipment.