Polymorph segmentation representation for medical image computing
Introduction
Segmentation, which is the process of delineating structures of interest, is a critically important task in many medical image computing and visualization workflows: it enables quantitative analysis inside structures such as organs and malignancies, facilitates therapy planning, and provides better understanding of patient anatomy. Various data structures are available to represent segmentation results. Unfortunately none of them are optimal for storage, analysis, and real-time visualization at the same time, so a trade-off is typically made to choose the most suitable representation for the main purpose of a specific application.
Most commonly, segmentation results are stored in 3D binary labelmap volume (shown in Fig. 1/A) where the value of each voxel indicates if that point is inside or outside the structure. This representation is the usual output of manual segmentation and preferred input for image processing and analysis algorithms. However, for 3D visualization of structures a surface model (Fig. 1/B) is more optimal, as it can be rendered by graphics cards efficiently. A surface model is a mesh consisting of tens or hundreds of thousands of connected triangles. Certain segmentation algorithms yield fractional labelmaps (Fig. 1/C) with voxels indicating probabilities [7] instead of a binary decision. For radiation therapy, the DICOM standard [11] requires the structures to be stored as “structure sets” - a series of planar contours (Fig. 1/D). Structure sets may also be represented as ribbons (Fig. 1/E), which can be easily computed from contours, indicate slice thickness, and make 3D shapes more recognizable.
In most workflows, it is not sufficient to use a single representation of segmentation data. An example for the need of multiple representations is the calculation of dose volume histograms (DVH) for radiation therapy, which are used for treatment plan evaluation and optimization. Targets to treat and organs to avoid are segmented by delineating their cross-section in multiple slices of the image, which results in a list of planar contours. Contours are then voxelized into labelmaps, so that they can be used as masks for dose distribution volumes and computing the histograms. For 3D visualization closed surfaces are shown, which can be created either from contours or labelmaps.
There are challenges involved in the design and implementation of storage and analysis of anatomical structures that must be addressed in all software applications:
- 1.
Conversion method selection: The need for conversion in order to prepare segmentation data for certain operations is something to be recognized by the user. Further, some details are often lost or altered during conversion. It is important to make users aware of these changes, while at the same time keep the software simple and easy to use.
- 2.
Provenance: The identity and origin of the structures and what they represent. In a typical research workflow not driven by a rigid processing pipeline it is up to the user to maintain the objects participating in the workflow. Often, this results in an unorganized set of objects with arbitrary names. When working with more than a few structures it quickly becomes unclear where each object originates from and what transformations they went through.
- 3.
Consistency: Representations may change after conversions (e.g. by manual editing), in which case the other representations of the same structure become invalid, and the data scene inconsistent. It is imperative to make sure that no invalid data is accessible at any time.
- 4.
Coherence: Structure sets typically correspond to the same entity (i.e., a patient), so when objects are stored in memory or disk, processed, or visualized it should be possible to manage them as a unified whole.
Extensive research has been performed towards developing fast and accurate conversion algorithms between representations ([2], [5]; Jian [8], [10], [13], [18], etc.) for various applications ranging from geography, through computer graphics to medical image computing. To our knowledge, however, no investigation has been done in terms of a complex workflow involving multiple data representations and the challenges identified earlier. Keeping the data in good shape is essential, especially if a human user is involved, or the workflow steps’ order is not fixed. By making these conversion methods more accessible and robust, the time spent in the workflow can be reduced considerably. An example is the dosimetry workflow by Alexander et al. [1], in which the time of analysis was reduced from hours to minutes.
This paper proposes a software system methodology and implementation for Polymorphic Segmentation Representation to facilitate dynamic management of multiple data formats representing the same structure in a way that addresses the identified challenges involved in the process. It aspires to open the way to rapid prototyping of more robust and user-friendly software for medical image analysis research and related clinical applications.
Section snippets
Computational methods and theory
A complex data structure called a segmentation object is proposed that unites different representations of segmented structures. It provides automatic conversion between representation types. New representation types can be dynamically added to the system. The software design is the result of a consensus reached through discussions with numerous researchers and developers participating in the Project Week open-source programming event series [9].
Core library
The basic storage and conversion features are implemented in a software library called PolySeg.1 It includes the segmentation object and segment classes, as well as those responsible for conversion. The core library is written entirely in C++ and is based on the Visualization Toolkit (VTK) [17]. VTK is well-established and one of the most widely used software libraries in the field of medical image computing: MITK [12], MeVisLab [16], medInria [24], NIfTI [3],
Samples of typical system
It has been shown through various applications that building on PolySeg yields robust and versatile software in a short development time. The following applications are based on 3D Slicer, being the first fully supported platform utilizing the PolySeg features. However, similar applications are possible to develop using PolySeg directly, or after integration to a similar medical image computing toolkit.
Hardware and software specifications
PolySeg is platform-independent, and is tested on Windows, MacOS, and Linux. Nightly automated testing ensures error-free operation on each operating system. The end-user application prototypes implemented using PolySeg and 3D Slicer work on the same three operating systems.
Hardware requirements are specific to the end-user applications built on top of PolySeg. However, the general requirement is a configuration that has the graphics capabilities to display and memory to hold the original
Mode of availability of the system and programs
The PolySeg library is distributed under the BSD 2-clause open-source license, which contains no restrictions on the use of the software. The source code and documentation are accessible freely on GitHub: https://github.com/PerkLab/PolySeg.
The implemented end-user applications are also under the BSD 2-clause license, and can be freely accessed at the following locations:
- •
Segment Editor within the 3D Slicer application: https://www.slicer.org
- •
Batch structure set conversion and open-source external
Acknowledgement
This work was supported by Cancer Care Ontario through the Applied Cancer Research Unit and Research Chair in Cancer Imaging grant, The Ontario Consortium for Adaptive Interventions in Radiation Oncology (OCAIRO) grant funded by the Ontario Research Fund, and the Research Software Program provided by CANARIE.
Conflict of interest statement
The authors have no relevant conflicts of interest to disclose.
References (27)
- et al.
3D Slicer as an image computing platform for the quantitative imaging network
Magn. Reson. Imaging
(2012) - et al.
Multi-atlas segmentation of biomedical images: a survey
Med. Image Anal.
(2015) - et al.
Validation of MRI to US registration for focal HDR prostate brachytherapy
Brachytherapy
(2017) - et al.
User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability
Neuroimage
(2006) - et al.
Streamlined open-source gel dosimetry analysis in 3D Slicer
Biomed. Phys. Eng. Express
(2018) Exploring and evaluating the consequences of vector-to-raster and raster-to-vector conversion
Photogramm. Eng. Remote Sens.
(1997)- et al.
A (sort of) new image data format standard: {NIfTI-1}
Neuroimage
(2004) - et al.
Optimal surface reconstruction from planar contours
ACM SIGGRAPH Comput. Graph.
(1977) - et al.
Comparing images using the Hausdorff distance
IEEE Trans. Pattern Anal. Mach. Intell.
(1993) - et al.
An accurate method for voxelizing polygon meshes
IEEE Symp.
(2014)
Increasing the impact of medical image computing using community-based open- access Hackathons : the NA-MIC and 3D Slicer experience
Surfaces from contours
ACM Trans. Graph.
Introduction to the DICOM standard
Eur. Radiol.
Cited by (48)
Validation of a scanning technique with minimal compression for measuring muscle volume with freehand 3D ultrasound
2024, Journal of BiomechanicsAlveolar cleft reconstruction utilizing a particulate autogenous tooth graft and a novel split-thickness papilla curtain flap — A retrospective study
2024, Journal of Cranio-Maxillofacial SurgeryQuantitative image signature and machine learning-based prediction of outcomes in cerebral cavernous malformations: Machine learning for cavernous malformations outcomes
2024, Journal of Stroke and Cerebrovascular DiseasesVT3D: a visualization toolbox for 3D transcriptomic data
2023, Journal of Genetics and Genomics