Introduction

Plain chest radiography remains a crucial tool for early detection of pulmonary tuberculosis (TB) and the monitoring of responses to TB treatment [1]. Chest X-rays (CXRs) have a high sensitivity in detecting pulmonary TB abnormalities, even in asymptomatic TB patients, especially when interpreted by experienced radiologists. Despite this, out of the estimated 10 million global TB cases in 2019, only 7.1 million were detected and reported [2]. As a result, although the global TB incidence rate and annual number of TB deaths has been steadily declining, it is not yet in line with the targets set out in the World Health Organization’s (WHO) End TB Strategy [2].

While advances in digital radiography technology have increased the quality of CXR images [3], limited access to these facilities and experienced radiologists remains a long-standing challenge, particularly in low-resource settings with a high TB burden [4]. However, recent advances in artificial intelligence (AI)-based computer-aided detection (CAD) systems have shown promising results in the automated interpretation of CXRs and detection of TB [5–7]. With acceptable accuracy, these CAD systems may help improve access to CXR reading for TB screening and contribute towards achieving WHO’s End TB strategy [2, 8]. However, there are limited number of studies in this area, most of which have methodological limitations, studied only one CAD software, with few screening data, and/or industry-funded [7–10]. Moreover, most studies used non-expert CXR interpreters and assessed an online CAD processing system or shared images with the CAD vendors and compared the performance against a suboptimal refence standard of a single sputum specimen tested with Xpert MTB/RIF which further highlights the need for independent and rigorous studies [8–12]. More recent investigations have focused on offline and multiple AI systems [13–15], but they remain few in number.

A global consultation, convened by WHO in 2016, concluded that additional evidence on the performance and use of available CAD systems for TB screening were required [16]. To address this need, the International Organization for Migration (IOM) and FIND entered into a research collaboration to conduct two parallel studies at their respective organizations, both evaluating the accuracy of TB CAD technologies. The studies were conducted independently of the developers, using similar study designs and analysis plans, but involving separate archives. The results of both studies have contributed to the updated WHO consolidated guideline TB screening [17].

Here we present the results of the IOM study, evaluating the diagnostic accuracy of three commercially available CAD systems for detecting TB in offline setting using an independent global archive of CXR images collected from multiple sites performing TB screening of migrants in different countries. against a microbiological (MRS) and radiological reference standard (RadRS).

Methods

Results

Study selection

A total of 2,910 cases were sampled (Fig 1): 589 culture-positive and 865 culture-negative from Sample 1, and 590 CXR suggestive of TB and 866 CXR not suggestive of TB from Sample 2. After 78 duplicates were excluded and 20 cases were rejected by the expert radiologists based on the CXRs, 2,812 (1781 RadRS positive and 1031 RadRS negative) cases were included in the RadRS analysis. Of these, 1,769 (533 MRS positive and 1236 MRS negative participants) were also included in the MRS analysis, of which 178 were RadRS positive and 1591 RadRS negative. The remaining 1,043 (190 RadRS positive and 853 RadRS negative) cases were not used for analysis against MRS because either they had > 14 days between the CXR exam and sputum collection, 177 (150 RadRS positive & 27 RadRS negative) or there was no available sputum culture or Xpert result, 866 (40 RadRS positive & 826 RadRS negative). Additionally, 207 participants with invalid CAD4TB scores (empty or negative values) were excluded from analysis for CAD4TB; Lunit and qXR had valid scores for all observations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of participants included in the analysis.

https://doi.org/10.1371/journal.pgph.0000402.g001

Baseline demographic and clinical characteristics

Baseline demographic and clinical characteristics of the whole study population (RadRS) and the population included in the analysis against MRS are presented in S4 Table and Table 1, respectively. Similarly, CXR findings and microbiological test results among RadRS and MRS population are presented in S5 Table and Table 2, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Baseline demographics and clinical characteristics (MRS analysis population).

https://doi.org/10.1371/journal.pgph.0000402.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Chest X-ray findings and microbiological test results (MRS analysis population).

https://doi.org/10.1371/journal.pgph.0000402.t002

In the MRS analysis population, more than half (60.5%) were male, most were young (36.2% were 15−35 years of age), 30.1% were MRS positive, and 89.9% had CXR suggestive of TB (RadRS positive). MRS-positive TB cases were reported more often among males (67.0%), at a younger age (44.5% were 15–35 years of age), and in those with TB symptoms compared to non-TB cases (3.6% vs 0.6%). Sputum smears were positive in 29.8% of MRS-positive cases, and abnormal CXR findings in 99%, 97.5% of which were CXR suggestive of TB (Table 1). However, 91% of the MRS negative cases also had abnormal CXR findings, 84.6% of which were CXR suggestive of TB. Only 4.7% of MRS TB cases had Xpert results in addition to cultures, and only two (0.1%) had discrepancies, one being culture-negative and Xpert-positive, and the other being culture-positive and Xpert-negative (Table 2).

In the RadRS analysis population (2812), similarly, most were male (55.3%), younger age group (15–35 year (45.4%), but only 1.2% had one or more TB symptoms and 10% had smear positive result (S4 Table). The 63.3% of RadRS analysis population had CXR findings suggestive of TB (RadRS positive cases. Of those, 32.3% were culture positive (S5 Table).

Overall diagnostic accuracy of index tests

The AUCs of the ROC curves, plotting the estimated sensitivity and specificity at each possible abnormality score against MRS, was highest with the Lunit system (0.85; 95% CI 0.83−0.87), followed by qXR (0.75; 95% CI 0.72−0.77) and then CAD4TB (0.71; 95% CI 0.68−0.73) (Fig 2A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Receiver operating characteristic curves of three CAD systems against microbiological and radiological reference standards.

Cases with no valid CAD4TB scores were excluded from the ROC analysis (n = 93 from the ROC against MRS; n = 207 against RadRS).

https://doi.org/10.1371/journal.pgph.0000402.g002

The ROC curves against RadRS showed greater AUCs for all CAD systems compared with the ROC curves against MRS analysis, being highest with CAD4TB (0.87; 95% CI 0.86−0.89) and Lunit (0.87; 95% CI 0.85−0.88), followed by qXR (0.81; 95% CI 0.80−0.83) (Fig 2B).

The point estimate for the sensitivity (98.3%; 95% CI 96.8−99.2%) and specificity (13.7%; 95% CI 11.8−15.7) of the expert radiologist is presented in the ROC against MRS for visual comparison of the radiologist performance with the performance of the CADs (Fig 2A), and it overlies along the line for Lunit.

Accuracy estimates of index tests

The sensitivity and specificity of the three CAD products computed at a range of thresholds against MRS showed the highest accuracy (a combination of sensitivity and specificity) for Lunit in all target categories, followed by qXR and CAD4TB. At a set sensitivity of 90%, specificity values were 54.5% (95% CI 51.7−57.3%) for Lunit, 32.4% (95% CI 29.8−35.1%) for qXR, and 23.0% (95% CI 20.6−25.5%) for CAD4TB. At a set specificity of 70%, sensitivity values were 81.4% (95% CI 77.9−84.6%) for Lunit, 67.9% (95% CI 63.8−71.9%) for qXR, and 61.7% (95% CI 57.3−65.9%) for CAD4TB.

At the sensitivity value achieved by expert radiologists (98.3% against MRS), the specificities of Lunit and qXR were 15.8% (95% CI 13.8−17.9%) and 12.0% (95% CI 10.2−13.9%), respectively, compared to 13.7% (95% CI 11.8−15.7) for the expert radiologist. CAD4TB had a lower specificity of 6.5% (95% CI 5.2−8.1%). At the specificity value achieved by expert radiologists (13.7% against MRS), the sensitivity was highest for Lunit (99.1%; 95% CI 97.8−99.7%), followed by qXR (97.7%; 95% CI 96.1−98.8%), then CAD4TB (95.9%; 95% CI 93.8−97.5%) (Table 3), compared to 98.3% (95% CI 96.8−99.2%) for the expert radiologist.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Accuracy estimates of index test at different sensitivity and specificity points (MRS analysis population).

https://doi.org/10.1371/journal.pgph.0000402.t003

At manufacturer-recommended thresholds, the three Lunit thresholds (0.15, 0.3 and 0.45) resulted in sensitivities of 84.2% (95% CI 80.9−87.2), 78.0% (95% CI 74.3−81.5) and 72.6% (95% CI 68.6−76.4), with specificities of 65.5% (95% CI 62.8−68.2), 76.3% (95% CI 73.8−78.6), and 82.6% (95% CI 80.4−84.7) respectively. However, the targets of the WHO TPP for a triage test were not met at any of the thresholds. For qXR, both threshold scores (0.55 and 0.75) resulted in lower sensitivities than Lunit: 56.7% (95% CI 52.3−60.9) and 26.8% (95% CI 23.1−30.8), with specificities of 78.8% (95% CI 76.4−81.1) and 93.0% (95% CI 91.4−94.3) respectively (Table 3).

Sensitivity and specificity estimates against RadRS are presented in S6 Table. At 90% sensitivity, specificity was highest for CAD4TB (54.6%; 95% CI 51.3−57.8), followed by Lunit (45.5%; 95% CI 42.4−48.6) and then qXR (37.5%; 95% CI 34.6−40.6). At 70% specificity, CAD4TB (84.1%; 95% CI 82.3−85.8) and Lunit (80.9%; 95% CI 78.9−82.7) had a higher sensitivity than qXR (70.0%; 95% CI 67.8–72.1) (S6 Table). At the manufacturer-recommended thresholds, both Lunit and qXR showed similar trends to the analysis against MRS, but typically with lower sensitivity and higher specificity values than against MRS (S6 Table). The three Lunit thresholds (0.15, 0.3 and 0.45) against RadRS resulted in sensitivities of 53.1% (95% CI.8−55.5), 42.9% (95% CI 40.6–45.2) and 36.6% (95% CI 34.3–38.8), with specificities of 98.4% (95% CI 97.5−99.1), 99.2% (95% CI 98.5−99.7), and 98.4% (95% CI 97.5−99.1) respectively. For qXR, both threshold scores (0.55 and 0.75) resulted in lower sensitivities than Lunit 35.0% (95% CI 32.8–37.2) and 13.8% (95% CI 12.2−15.4), and specificities of 96.6% (95% CI 95.3−97.6) and 99.7% (95% CI 99.2−99.9) respectively (S6 Table).

Image processing errors were noticed for 178 CXR images after processing by Lunit, in which the images were inverted from the original negative image (bones white) to positive (bones black). The sensitivity and specificity values, as well as the AUC of Lunit with and without those cases included, were unaffected by these processing errors.

Discussion

This study is one of the first comprehensive studies evaluating CAD systems independent of the CAD developers in a population screened for TB using both culture results and expert radiologist assessments as reference standards. The findings from the study demonstrated that the three CAD systems (Lunit, CAD4TB, qXR) have comparable diagnostic accuracy in detecting TB on CXR when used for TB screening and may perform comparably to that of expert radiologists, with Lunit performing better than both qXR and CAD4TB against MRS and CAD4TB and Lunit performing better than qXR against RadRS. However, none of the CAD systems reached the minimum performance requirements of the WHO triage TPP (90% sensitivity and 70% specificity) [19], in contrast to previously published findings by Khan et al. [13].

The finding that i) Lunit performed best against MRS and ii) CAD4TB performed best against RadRS, shows that the CADs performance can vary by the reference standard used, and may indicate that Lunit is better at detecting CXR findings suggestive of active TB disease, which tend to be culture-positive, while CAD4TB may better detect CXR findings suggestive of old, healed TB that can be identified by radiologists, but tend to be culture-negative. This finding could also reflect the methodology used in the deep machine learning of the CAD product algorithms, e.g., mainly training the software against RadRS versus MRS. The better performance of CAD4TB against RadRS than MRS is also supported by the results of a previous study by Fehr et al. [11].

The low specificity of the CADs at a set sensitivity of 90% against MRS is similar to the expert radiologist and likely is a result of the selection of our study population in whom sputum samples were usually only collected when the initial CXR reading was suggestive of TB or when there was a clinical suspicion of TB. However, also the intrinsic nature of sputum culture, CXR, and CXR signs of TB may be an explanation. Culture analysis detects TB in cases with detectable bacteria in the sputum. As such, it measures the sensitivity of detecting active TB disease, whereas, old, healed pulmonary lesions detected by CXR can be culture-negative and are considered false-positive in the analysis against MRS. Additionally, CXR signs of TB are not specific to TB only, thereby reducing the estimated specificity. Therefore, CXR is recommended for screening but not as a confirmatory diagnostic tool (i.e., a positive CXR TB screening result should be used as criteria for further confirmatory testing, such as sputum cultures, and not for a treatment decision). Nevertheless, for a screening tool the benefit of high sensitivity may outweigh the limitations of a lower specificity. Both Lunit and qXR had relatively lower sensitivity and specificity at all manufacture provided thresholds, though Lunit performed with relatively higher sensitivity while qXR achieved higher specificity. As such, the sensitivity and specificity thresholds of the CADs that correspond with expert radiologist assessments (98.3% and 13.7%, respectively), may be potential candidates for the selection of optimal thresholds for operational use.

Subgroup analyses showed that the performance of CADs can vary among some population demographic and clinical characteristics. All CAD systems performed worse in participants with a history of TB, something which has also been observed in previous studies [13–15]. This is to be expected, as healed TB can leave residual CXR changes, which usually are classified as TB findings on CXR but can lead to negative microbiological test results. CAD4TB performed worse in participants with smear-negative results, in line with the findings of Khan et al. [13]; CAD4TB, moreover, performed worse in cases where additional CXR views were requested by the expert radiologist. Again, these results are not surprising, as smear-positive cases may have obvious CXR abnormalities that can be easily detected, and the absence of a request for additional views may indicate that the initial CXR was of good quality and/or there were no suspicious CXR findings. However, this conclusion was significant only for CAD4TB, while a similar although not statistically significant trend was observed for Lunit and qXR.

Additional trends were observed in the other subgroup analyses. While overlapping CI values indicate that these findings should be interpreted with caution, it appeared that the CAD systems performed less well in females, participants with no TB symptoms, HIV-positive participants, those with an ‘immigrant’ status compared with those classified with a ‘refugee’ status, and in older participants (for Lunit only). Other studies have also reported that CAD performance can be significantly impacted by sex and age [13]. The differences in CAD performance among different subgroups indicates that population characteristics should be taken into consideration before implementation.

There are some limitations to this study that should be considered. Firstly, our study relied on retrospective data from a routine migration screening program. As such participants received sputum smear and culture tests during the initial TB screening only when the initial CXR reading was suggestive of TB or there was a clinical suspicion of TB. Therefore, sputum culture testing was not performed for most participants with normal CXRs or CXR findings suggestive of non-TB and were not included in the MRS analysis. This likely resulted in spuriously increase in sensitivity and lower specificity readings for the CADs and expert radiologists, as would be expected from an unselected population. Moreover, as TB cases were overrepresented for both the MRS and RadRS analyses due to the sampling strategy employed in this study, the dataset may not be representative of all people presenting for TB screening but is instead a subset of those who had a higher suspicion of TB and therefore underwent sputum examination. This might have affected the overall accuracy estimates, albeit to a similar extent for all three CAD systems and the expert radiologists, thus, we believe the comparison between the accuracy of CADs versus expert radiologists holds true. Additionally, 20 participants with no radiologist assessment were excluded from the analyses, as well as 207 images from the CAD4TB analysis due to invalid score results. The reason for the invalid scores with the CAD4TB system was unknown, but it could be because the software quality control rejected unacceptable or poor images without requiring further investigation. Although the number of excluded results is small compared with the size of the whole dataset, it is possible that the characteristics of the cases excluded may have been different from those that were included.

Although the study did not systematically evaluate quality control measures of the CADs, some issues were observed during the automated interpretation of the CXRs by the CAD systems. Some CXR projections other than the PA CXRs, such as lateral and lordotic CXR views (which are unsupported by the CADs) or CXRs with image quality issues, were not always flagged by the systems.

The study also did not assess the operational performance of the CADs such as the processing time, technical issues, and troubleshooting responses, infrastructure needs, comparison of offline and online use of the CAD product, cost-effectiveness, or other related matters. In addition, since the study was conducted new versions of the CAD systems have been released and other CAD systems have entered the market [21], which may necessitate further evaluation.

Based on the findings of this study, combined with those of the parallel study conducted by FIND [22], CAD systems may be considered viable as a tool for automated CXR interpretation with regard to TB detection in screening programs, particularly in remote, and/or high TB burden places where there are limited resources and access to expert radiologists. The use of CAD systems in these areas may even have a wider application and contribute to increase the global TB detection rate. Further to these, and other, findings, WHO has recently released consolidated guidelines on tuberculosis recommending that CAD may be used in place of human readers for interpreting digital CXR for TB screening in individuals aged 15 years and older [17]. Another role of CADs, even in places where expert radiologists are available, may be their use for internal quality control monitoring of CXRs complementary to radiologist assessments.

Nevertheless, further studies may be required to investigate the accuracy of CADs in detecting non-TB-significant findings, such as lung cancer or bone lesions as well as the different specific CXR findings suggestive of TB, better address the performance of CADs in the different population subgroups, the way the CADs address image quality issues that might necessitate repeat CXRs or additional views by radiologists, how the CADs handle non-PA CXRs, and non-complied age requirements for specific systems. Likewise, prospective studies are needed to address the operational use of the CAD systems [23], including choice of the CAD system and version, compatibility with X-ray machine, accepted image format, need for validation, integration into existing workflow and patient registration systems, feasibility of online or offline use of the software, and technical requirements, as well as the selection of optimal thresholds for the intended use.

In conclusion, the results of this study demonstrated the comparability of the accuracy of three CAD systems for CXR interpretation with regard to TB screening, which may broadly perform similar to that of an expert radiologist. Additionally, the study has demonstrated that the performance of the CAD systems can vary by population demographic and clinical characteristics as well as the reference standard used. As such, these tools may provide viable options for use in TB screening programs to increase TB detection, especially in low resource areas where there may be no available expert radiologists. However, further studies are needed to better address CAD performance in specific population subgroups or different CXR TB findings, to assess other operational and technical factors necessary for proper operational implementation, and to evaluate novel CAD products coming to the market.

Supporting information

Acknowledgments

We would like to thank all the study participants who took part in the IOM health assessment screening and included in the study, the three CAD software companies, Delft Imaging, Qure.ai and Lunit INSIGHT, for installing the CAD products on the IOM server and providing the technical support and troubleshooting, CDC for approving the use of the data and CXR images from the IOM health assessment TB screening of migrants bound to the United States, and Dr. Mary Naughton and Dr. Drew Posey from CDC for assisting in the process, Dr. Cecily MILLER and Dr. Dennis FALZON from the WHO TB programme for their technical collaboration during the study and allowing us to present results of the studies to the WHO TB guideline development group meeting, IOM Legal counsel for approving to use the Data and CXR images of migrants, as well as Dr Fiaz Ahmad Khan from McGill university for his coordination of the Ethical approval. Additionally, we thank our IOM colleague Mr. Rommel Cordero for assisting the technical support and troubleshooting, our IOM Teleradiology Radiologists Dr. Ethel Enriquez-Alas, Dr. Sthennette Jerusalem for re-interpreting the CXRs included in the study, Dr. Lena Maria Ablis-Sun for assisting in quality control review of the discrepant cases and finalizing the CXR report, Ms. May Antonnette Lebanan for her technical guidance during the data analysis, and Dr. Paul Douglas, and Mrs. Jacqueline Weekers and Mr. Enrico Ponziani for the overall support and guidance.

Disclaimer: The findings and conclusions of this article are those of the authors and do not necessarily represent the official position of the US Government or CDC. References in this manuscript to any specific commercial CAD products, process, service, manufacturer, or company do not constitute endorsement or recommendation by the US Government or CDC.