Iterative fully convolutional neural networks for automatic vertebra segmentation and identification
Introduction
Segmentation and identification of the vertebrae is often a prerequisite for automatic analysis of the spine, such as detection of vertebral fractures (Yao et al., 2012), assessment of spinal deformities (Forsberg et al., 2013), or computer-assisted surgical interventions (Knez et al., 2016). Automatic spine analysis can be performed with a large variety of tomographic scans, including dedicated spine scans but also scans of the neck, chest or abdomen that incidentally cover part of the spine. A generic vertebra segmentation algorithm therefore needs to be robust with respect to different image resolutions and different coverages of the spine. This especially means that no assumptions should be made about the number of visible vertebrae and their anatomical identity, i.e., to which section of the spine they belong. Vertebra segmentation is therefore essentially an instance segmentation problem with an a priori unknown number of instances (i.e. vertebrae). However, in contrast to generic instance segmentation the individual instances are not independent of each other. The instances are known to be located in close proximity to each other in the image, forming together the vertebral column. We propose to approach vertebra segmentation with an instance segmentation algorithm that explicitly incorporates this prior knowledge to locate instances, but that makes no further assumptions.
Approaching vertebra segmentation as an instance segmentation problem entails treating all vertebrae as instances of the same class of objects. However, an anatomical identification of the segmented vertebrae is often also needed, for instance, for further analysis steps or for reporting purposes. Especially in images originally not intended for spine imaging, anatomical labeling of the vertebrae can be challenging due to variations in the field of view. These variations lead to variable coverage of the spine and also of structures that provide anatomical cues for identification of the vertebrae, such as the ribs or the sacrum. Additionally, neighboring vertebrae often have similar shape and appearance so that independent labeling of each vertebra may result in mistakes. Vertebra identification therefore requires a global rather than a per-instance approach to ensure an overall plausible, anatomically correct labeling.
Another challenge inherent to an instance segmentation approach is the identification of partially visible instances. While occlusion is a typical problem in two-dimensional but not in three-dimensional images, some vertebrae may be only partially visible due to the limited field of view of the scan. If these incompletely visible vertebrae are included in subsequent analyses that are based on the obtained vertebra segmentations, such as measurement of vertebral heights for detection and classification of vertebral compression fractures (Grigoryan et al., 2003), their results may be unreliable. Therefore, incompletely visible instances need to be either ignored or explicitly identified as incomplete so that they can be excluded from subsequent analyses.
In this paper, we propose an iterative instance-by-instance segmentation approach for vertebra segmentation based on a fully convolutional neural network. This network performs vertebra detection, segmentation, anatomical identification and classification of their completeness concurrently and therefore presents an entirely supervised approach that can be trained end-to-end. While we propose to attempt a per-instance identification of the individual vertebrae together with the segmentation, the labeling is subsequently adjusted taking all segmented vertebrae into account. In contrast to previous approaches, the presented method can be used for any imaging modality, any field of view and any number and type (cervical, thoracic, lumbar) of visible vertebrae because it avoids explicit modeling of shape and appearance of the vertebrae and the vertebral column. We evaluate these claims using a diverse selection of datasets, including scans from different modalities (CT and MR), various fields of view, cases with severe compression fractures and a particularly challenging set of low-dose chest CT.
Section snippets
Related work
While a few other methods have been published that address both vertebra segmentation and identification (Klinder, Ostermann, Ehm, Franz, Kneser, Lorenz, 2009, Kelm, Wels, Zhou, Seifert, Suehling, Zheng, Comaniciu, 2013, Chu, Belavỳ, Armbrecht, Bansmann, Felsenberg, Zheng, 2015, Suzani, Rasoulian, Seitel, Fels, Rohling, Abolmaesumi, 2015, Sekuboyina, Valentinitsch, Kirschke, Menze), the majority of methods in the literature focused on one of these problems. The existing literature is therefore
Methods
We propose a vertebra segmentation and identification method based on a single fully convolutional neural network (FCN) that performs multiple tasks concurrently. In contrast to existing methods, this avoids a multi-stage process with successive instance detection and segmentation, or segmentation and instance separation steps. Other existing generic instance segmentation methods with 2D deep neural networks often do not generalize well to 3D image volumes because they analyze the entire image
Datasets
We trained and evaluated the method with five sets of CT and MR scans that visualize the spine. Reference segmentation masks for four of these datasets are publicly available, which allowed for a comparison with other publications that used the same data. Examples of images from the datasets are shown in Fig. 3.
The thoracolumbar spine CT dataset consists of 15 dedicated spine CT scans that visualize all thoracic and lumbar vertebrae. It was originally used for the spine segmentation challenge
Experiments and results
We trained modality-specific instances of the network adjusted to the different ground truths, i.e., to perform vertebra segmentation in CT and vertebral body segmentation in MR. The CT training set consisted of 60 scans, of which 10 were thoracolumbar spine CT, 10 lumbar spine CT scans with compression fractures and 40 NLST scans. The CT evaluation set consisted of 30 scans, of which 5 were thoracolumbar spine CT scans, 5 lumbar spine CT with compression fractures, 10 NLST scans, and 10 normal
Discussion
This paper demonstrates that fully convolutional neural networks, which have been widely used for semantic segmentation (Litjens et al., 2017), are also capable of learning a complex instance segmentation task. Vertebra segmentation performed instance-by-instance required the network to learn to infer from an additional memory input which vertebra to segment and to ignore other vertebrae. Additionally, the same network was able to perform multiple tasks concurrently, namely vertebra
Acknowledgements
We would like to thank the organizers of the CSI 2014 spine segmentation challenge, the Laboratory of Imaging Technologies at the University of Ljubljana and the authors of the MR dataset for making scans and reference segmentations publicly available. We are furthermore grateful to the United States National Cancer Institute (NCI) for providing access to NCI’s data collected by the National Lung Screening Trial. The statements contained in this publication are solely ours and do not represent
References (62)
- et al.
Automatic segmentation of vertebral contours from CT images using fuzzy corners
Comput. Biol. Med.
(2016) - et al.
Interactive live-wire boundary extraction
Med. Image Anal.
(1997) - et al.
Multi-modal vertebrae recognition using transformed deep convolution network
Comput. Medical Imag. Graph.
(2016) - et al.
Automatic inference of articulated spine models in CT images using high-order markov random fields
Med. Image. Anal.
(2011) - et al.
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Med. Image Anal.
(2017) - et al.
Automated model-based vertebra detection, identification, and segmentation in CT images
Med. Image Anal.
(2009) - et al.
Automated landmarking and labeling of fully and partially scanned spinal columns in CT images
Med. Image. Anal.
(2013) - et al.
A hierarchical 3D segmentation method and the definition of vertebral body coordinate systems for QCT of the lumbar spine
Med. Image. Anal.
(2006) - et al.
V-net: Fully convolutional neural networks for volumetric medical image segmentation
International Conference on 3D Vision
(2016) - et al.
Detection of vertebral body fractures based on cortical shell unwrapping
International Conference on Medical Image Computing and Computer-Assisted Intervention. LNCS, 7512
(2012)