Evaluation of automatic neonatal brain segmentation algorithms: The NeoBrainS12 challenge
Graphical abstract
Introduction
Brain development in neonates may be affected by preterm birth. It is known that intra-uterine growth restriction, inflammation and hypoxia–ischemia will impede brain development (Tolsa et al., 2004, Volpe, 2009). These factors may affect brain growth and thus may result in altered brain tissue volumes in preterm infants. Analyses of these volumes have identified several risk factors and have provided important information regarding impaired brain development (Thompson et al., 2006). Nowadays brain tissue volumes at term equivalent age (TEA) are considered predictive markers of long-term neurodevelopmental performance (Peterson et al., 2003, Inder et al., 2005).
Accurate brain tissue segmentation is a prerequisite for obtaining volumetric measurements. Thus far, a number of different algorithms have been presented but comparing their performance reliably has not proven feasible.
The algorithms have been evaluated with images made using a variety of acquisition protocols involving differences regarding magnetic field strength, scan orientation and spatial resolution. For example, several methods were applied to images acquired at 1.5T scanners (Cardoso et al., 2013, Anbeek et al., 2008, Weisenfeld and Warfield, 2009), while other used 3T scanners (Gui et al., 2012b, Prastawa et al., 2005, Xue et al., 2007), and segmentations were designed using axial (Anbeek et al., 2008) or coronal (Gui et al., 2012b, Cardoso et al., 2013) image acquisitions.
In addition, the algorithms were evaluated using different criteria. While the quality of the segmentations was generally evaluated using overlap measures, notably the Dice coefficient (DC), the definition of the evaluated tissue classes varied. Since obtaining a manually provided reference standard for brain tissue segmentations has proven extremely time-consuming and cumbersome, very few methods have been evaluated on complete scans; frequently the performance of methods was assessed on a limited subset of scan sections. In addition, the published algorithms have not consistently segmented the same tissue (sub-)types. For example, several algorithms distinguished between cortical and central grey matter (e.g. Gui et al., 2012b, Weisenfeld and Warfield, 2009, Anbeek et al., 2008), whereas others combined these structures into one segment (Prastawa et al., 2005). Similarly, several methods identified myelinated white matter (e.g. Weisenfeld and Warfield, 2009), while others did not (e.g. Anbeek et al., 2008).
The NeoBrainS12 study, providing three different image sets of preterm born infants, was set up to enable a reliable comparison of different brain tissue segmentation methods. These sets are (i) axial scans acquired at 40 weeks corrected age, (ii) coronal scans acquired at 30 weeks corrected age and (iii) coronal scans acquired at 40 weeks corrected age. Each of these three sets consists of three T1- and T2-weighted MR images of the brain acquired with a 3T MRI scanner. The task was to segment cortical grey matter (CoGM), non-myelinated white matter (UWM) and myelinated white matter (MWM), brainstem (BS), basal ganglia and thalami (BGT), cerebellum (CB), and cerebrospinal fluid in the ventricles (CSFVent) and in the extracerebral space (CSFEcS) separately. Any team could upload the results and all segmentations were evaluated in the same way. This paper presents the results of eight participating teams.
Section snippets
Data
The NeoBrainS12 data set consists of three different image sets of premature infants: (i) axial scans acquired at 40 weeks corrected gestational age, (ii) coronal scans acquired at 30 weeks corrected gestational age and (iii) coronal scans acquired at 40 weeks corrected gestational age. All scans were made as part of standard clinical practice at the neonatal intensive care unit of the University Medical Center Utrecht, The Netherlands. In these images, no brain pathology was visible. Furthermore,
Manual annotations
All eight tissue classes were manually delineated (CoGM, UWM, BS, CB, CSFVent, CSFEcS, BGT, and MWM). Each tissue class was segmented separately. Manual reference segmentations have been defined on T2-weighted images. They were generated using in-house developed software. Each voxel was assigned to only one of the eight tissue classes by mouse painting. The labeling was indicated by color overlay, where one color uniquely represented each tissue (Fig. 1). During manual segmentation, observers
Evaluation
Automatically obtained segmentations of each tissue were compared with the manually obtained reference standard. The automatic segmentations were evaluated using the Dice coefficient (DC), Hausdorff distance (HD) and mean surface distance (MSD), i.e. the mean distance between the automatic and reference segmentations along their boundaries. The DC, HD and MSD values of a tissue were computed as the average values obtained over the provided number of scans (patients) within a set.
In addition, an
Methods
Below, a brief description of the algorithms participating in the NeoBrainS12 challenge is given. Links to extensive descriptions of the methods are available in the Results section of the NeoBrainS12 webpage.1 Table 2 provides a list of the data segmented by each method, images used in segmentation, application of inhomogeneity correction and total number of segmented tissues.
Results and discussion
Teams were free to choose which set of images they would like to segment. Four methods, i.e. methods A, B, C and D, have been applied to all sets, while the remaining four methods (E, F, G and H) have been evaluated with only one of the sets. Similarly, five methods (methods B, C, D, F and G) have segmented all defined tissue classes, whereas method A segmented all tissues except MWM, method E segmented CSFEcS and CSFVent as one tissue class, and method H only segmented UWM, CoGM and CSFEcS.
Conclusions
Evaluation of eight methods for neonatal brain segmentation in cortical grey matter, non-myelinated and myelinated white matter, brainstem, basal ganglia and thalami, cerebellum, cerebrospinal fluid in the ventricles and cerebrospinal fluid in the extracerebral space on publicly available neonatal MRI scans has been presented. The methods were evaluated on axial scans acquired at 40 weeks corrected gestational age, and coronal scans acquired at 30 weeks and 40 weeks corrected gestational age. The
Acknowledgements
This study includes images of infants participating in the Neobrain study (LSHM-CT-2006-036534).
The authors from King’s College London would like to acknowledge the funding/support provided by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Guy’s and St. Thomas’ NHS Foundation Trust and King’s College London. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
The authors from University of
References (39)
- et al.
Probabilistic brain tissue segmentation in neonatal magnetic resonance imaging
Pediatr. Res.
(2008) - et al.
Unified segmentation
NeuroImage
(2005) - et al.
Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain
Med. Image Anal.
(2008) - et al.
The morphological approach to segmentation: the watershed transformation
- et al.
Adaptive neonate brain segmentation
Int. Conf. Med. Image Comput. Comput.-Assisted Inter.
(2011) - Cardoso, M.J., Melbourne, A., Kendall, G.S., Robertson, N., Marlow, N.O.S., 2012. Outlier rejection for adaptive...
- et al.
AdaPT: an adaptive preterm segmentation algorithm for neonatal brain MRI
NeuroImage
(2013) - et al.
Active contours without edges for vector-valued images
J. Vis. Commun. Image Represent.
(2000) - Chiţaˇ, S., Benders, M.J., Moeskops, P., Kersbergen, K.J., Viergever, M.A., Išgum, I., 2013. Automatic segmentation of...
- et al.
Nonlinear anisotropic filtering of MRI data
IEEE Trans. Med. Imaging
(1992)
Morphology-driven automatic segmentation of MR images of the neonatal brain
Med. Image Anal.
Abnormal cerebral structure is present at term in premature infants
Pediatrics
elastix: a toolbox for intensity-based medical image registration
IEEE Trans. Med. Imaging
A dynamic 4D probabilistic atlas of the developing brain
NeuroImage
Retrospective correction of MR intensity inhomogeneity by entropy minimization
IEEE Trans. Med. Imaging
Multimodality image registration by maximization of mutual information
IEEE Trans. Med. Imaging
Cited by (86)
MimicNet: Mimicking manual delineation of human expert for brain tumor segmentation from multimodal MRIs
2023, Applied Soft ComputingDevelopment of cortical folds in the human brain: An attempt to review biological hypotheses, early neuroimaging investigations and functional correlates
2023, Developmental Cognitive NeuroscienceBrain segmentation in patients with perinatal arterial ischemic stroke
2023, NeuroImage: ClinicalHarmonized neonatal brain MR image segmentation model for cross-site datasets
2021, Biomedical Signal Processing and ControlAutomatic segmentation and location learning of neonatal cerebral ventricles in 3D ultrasound data combining CNN and CPPN
2021, Computers in Biology and Medicine