Elsevier

Pattern Recognition Letters

Volume 140, December 2020, Pages 10-17
Pattern Recognition Letters

From 3D to 2D: Transferring knowledge for rib segmentation in chest X-rays

https://doi.org/10.1016/j.patrec.2020.09.021Get rights and content

Highlights

  • Traditional chest X-ray rib segmentation algorithms lack generalization capabilities.

  • We propose a novel pipeline for rib segmentation without using any label from chest X-rays.

  • This paper introduces a new label set for the JSRT dataset in the task of rib segmentation.

  • A new digitally reconstructed radiograph dataset is publicised in this manuscript.

  • Conditional domain adaptation shown to be more robust for domain generalization.

Abstract

Chest X-rays are the most common type of biomedical radiologic exam, being widely adopted for the diagnosis of a myriad of illnesses in the thoracic region. Computed Tomography – even though being more expensive and rare – is also a useful tool for the detection of several illnesses and surgery planning, providing volumetric information. This paper proposes a methodology aiming to leverage the larger amounts of spatial information and lack of occlusion in tomographic images to aid in the rib segmentation of 2D X-ray images by means of Domain Adaptation. We perform extensive quantitative and qualitative experiments to test the capabilities of this methodology in segmenting ribs in 7 X-ray datasets with distinct visual features, using 6 different metrics and without any use of rib segmentation labels from the target image sets. In order to encourage reproducibility, all data and code used in this research is publicly available online, including a new 2D Digitally Reconstructed Radiograph generated from tomographic data and a new pixel-level label map for the JSRT Chest X-ray dataset. We also publicize our generalizable pretrained models for both rib segmentation in Chest X-rays and lung field segmentation in Digitally Reconstructed Radiographs. Results show that the proposed pipeline outperforms shallow rib segmentation baselines in almost all quantitative metrics and produce higher fidelity pixel-map predictions than simply using the pretrained Neural Networks on the flattened 3D data, mainly in datasets where domain shift is more pronounced. The use of Conditional Domain Adaptation also allows the method to perform inference on all 7 X-ray datasets using one single model, achieving over 0.856 of AUC on OpenIST and 0.934 of AUC on JSRT, with Dice scores of 0.68 and 0.69 in these two datasets, respectively.

Introduction

Deep Neural Networks (DNNs) have achieved state-of-the-art results in both computer vision and biomedical image analysis [1]. DNNs are powerful overcomplete statistical models that can learn to extract features from unstructured data such as image, audio, video and text. These models can be understood as ensembles of perceptrons organized in layers with increasing semantic capabilities, being able to extract and select features and to perform inference conjointly. In general, deeper models are able to encode information with higher semantics, while shallower models can only optimize low-level semantic information. Convolutional Neural Networks (CNNs) [2] and their variants [3], [4] are the most popular architectures used for performing inference (i.e. classification, segmentation and detection tasks) over images, including biomedical settings.

Despite recent efforts [5], [6], [7] in acquiring large labeled datasets – mainly for diagnosis, that is, classification tasks – most biomedical image domains suffer from a lack of labeled data. Therefore, it is highly desirable to acquire the most amount of knowledge possible with the few labeled data available in the literature, while also using the vast amounts of unlabeled data present in some domains. Aiming to lessen the requirements for labeled data in visual recognition, Domain Adaptation (DA) [8], [9], [10], [11] is the research area that comprises the theoretical background and methods for knowledge transfer between distinct tasks and/or data.

Chest X-rays (CXRs) are the most common type of radiological exam acquired nowadays, mainly due to their ability to aid in the detection and diagnosis of several kinds of ailments [6]. Several diverse health conditions such as pulmonary nodules [12], tuberculosis [13], pulmonary effusion, pneumonia and cardiomegaly [6], as well as bone fractures, can be assessed by CXRs in a quick and radiation-efficient exam. With the advent of large labeled datasets [6], [7], automation efforts have been proposed for aiding in the diagnosis of most of these illnesses. Computed Tomography (CT) exams yield volumetric images that allow physicians to perform 3D analysis of the thorax, but require more expensive hardware and submit the patient to between one and two orders of magnitude larger doses of radiation.

As argued by Zhang et al. [14] the understanding of anatomical objects in CXRs is useful for several clinical applications, such as pathological diagnosis, treatment evaluation, surgical planning and as an automated preprocessing step for Computer-Aided Detection (CAD) systems. Van Ginneken et al. [15], [16] point that the delineation of rib borders is part of this anatomical registration process whose automation can help physicians by providing a frame of reference for the location of abnormalities [14], [16] and help surgery planning. However, computerized analysis is still the largest beneficiary from algorithms for rib cage detection, being useful to mitigate both false positives [17] and false negatives [18] in nodule or rib fracture detection, which may also indicate other issues as damage to lung tissue, hemorrhage signs of osteoporosis or even an underlying cancer [19]. For instance, Austin et al. [18] report that between 82% and 95% of undetected lung cancers in CXRs were obscured by foreground bones such as ribs or clavicles. Yet, mainly due to the great burden involved in pixel-level annotations, there is only a tiny amount of publicly available labeled samples for the task of rib segmentation in CXRs, as further discussed in Section 3.3.

Given the problems related to acquiring pixel-level labels for rib segmentation and the usefulness of these data, the main contribution of this paper is a pipeline based on Conditional Domain Adaptation [20] for rib cage segmentation. Secondary contributions of this work include validating the use of volumetric data for CXR bone segmentation, presenting a new rib segmentation label set for the JSRT dataset [12], and defining a standard quantitative and qualitative comparison procedure for rib cage segmentation.

The remaining sections of this paper are organized as follows. Section 2 describes the current state of the literature on rib segmentation and suppression methods. Section 3 presents the proposed method for rib segmentation from CT data, while also discussing the experimental setup (i.e. datasets, metrics, etc) used in the tests. Section 4 shows the quantitative and qualitative results yielded by the experimental setup, comparing them to the baselines of rib segmentation and with pretrained DNNs. At last, Section 5 concludes the paper with our final remarks and future works.

Section snippets

Related work

This section presents the current literature on rib segmentation and suppression methods. It also describes the basis of Unsupervised Image-to-Image Translation, which is the theoretical basis for the proposed method.

Methodology

In this section, we present the proposed methodology for rib cage segmentation in CXRs by using Unsupervised Domain Adaptation (UDA) from DRRs. Our method leverages the capabilities of Conditional DA [20] to transfer the knowledge learned from synthetically flattened CT-scans to 2D CXRs. Sections 3.4 and 3.3 present a standardized set of metrics for rib segmentation evaluation and the datasets used in this research, respectively.

Results and discussion

Before discussing the rib cage segmentation results, we present a subset of segmentation predictions by CoDALungs in DRRs from LIDC-IDRI (Fig. 2), also showing the poor lung segmentation generalization achieved by the baselines. These intermediate results are important because inaccurate segmentation predictions on the DRRs should result in poor bone mask filtering for separating the ribs from other bones in the MIP outputs. Most segmentations from CoDALungs are near perfect in identifying lung

Conclusion

In this paper we presented a novel methodology for UDA applied to the problem of rib segmentation using Conditional Domain Adaptation [20]. The proposed pipeline uses higher dimensional 3D data to acquire two sets of flattened 2D images: DRRs that visually resemble real CXRs – serving as training samples for rib segmentation; and bone segmentation semantic maps that can be curated in order to become pixel-level rib segmentation labels.

We also proposed a novel evaluation procedure for rib

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Authors would like to thank NVIDIA for the donation of the GPUs that allowed the execution of all experiments in this paper. We also thank CAPES, CNPq (424700/2018-2 and 311395/2018-0), and FAPEMIG (APQ-00449-17 and APQ-00519-20 -- CAD-COVID-19 Project) for the financial support provided for this research.

References (39)

  • V.M. Patel et al.

    Visual domain adaptation: a survey of recent advances

    IEEE Signal Process. Mag.

    (2015)
  • L. Shao et al.

    Transfer learning for visual categorization: asurvey

    IEEE Trans. Neural Netw. Learn. Syst.

    (2015)
  • J. Zhang, W. Li, P. Ogunbona, Transfer Learning For Cross-Dataset Recognition: A Survey,...
  • J. Shiraishi et al.

    Development of a digital image database for chest radiographs with and without a lung nodule receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules

    Am. J. Roentgenol.

    (2000)
  • S. Jaeger et al.

    Two public chest X-ray datasets for computer-aided screening of pulmonary diseases

    Quant. Imaging Med. Surg.

    (2014)
  • Y. Zhang et al.

    Task driven generative modeling for unsupervised domain adaptation: application to X-ray image segmentation

    International Conference on Medical Image Computing and Computer Assisted Intervention

    (2018)
  • B. van Ginneken et al.

    Automatic delineation of ribs in frontal chest radiographs

    Medical Imaging 2000: Image Processing

    (2000)
  • B. Van Ginneken et al.

    Computer-aided diagnosis in chest radiography: asurvey

    IEEE Trans. Med. Imaging

    (2001)
  • E. Soleymanpour et al.

    Fully automatic lung segmentation and rib suppression methods to improve nodule detection in chest radiographs

    J. Med. Signals Sens.

    (2011)
  • Cited by (0)

    Handle by Associate Editor-in-Chief Gabriella Sanniti di Baja, PhD.

    View full text