Polymorph segmentation representation for medical image computing

https://doi.org/10.1016/j.cmpb.2019.02.011Get rights and content

Highlights

  • A software library called PolySeg is developed to facilitate dynamic management of segmented structures in different representations.

  • PolySeg provides automatic conversions between numerous representations (e.g. labelmap, surface) to support analysis workflows or interactive editing.

  • Segment Editor in the open-source 3D Slicer platform provides easy-to-use manual and semi-automatic segmentation features.

Abstract

Background and objective

Segmentation is a ubiquitous operation in medical image computing. Various data representations can describe segmentation results, such as labelmap volumes or surface models. Conversions between them are often required, which typically include complex data processing steps. We identified four challenges related to managing multiple representations: conversion method selection, data provenance, data consistency, and coherence of in-memory objects.

Methods

A complex data container preserves identity and provenance of the contained representations and ensures data coherence. Conversions are executed automatically on-demand. A graph containing the implemented conversion algorithms determines each execution, ensuring consistency between various representations. The design and implementation of a software library are proposed, in order to provide a readily usable software tool to manage segmentation data in multiple data representations. A low-level core library called PolySeg implemented in the Visualization Toolkit (VTK) manages the data objects and conversions. It is used by a high-level application layer, which has been implemented in the medical image visualization and analysis platform 3D Slicer. The application layer provides advanced visualization, transformation, interoperability, and other functions.

Results

The core conversion algorithms comprising the graph were validated. Several applications were implemented based on the library, demonstrating advantages in terms of usability and ease of software development in each case. The Segment Editor application provides fast, comprehensive, and easy-to-use manual and semi-automatic segmentation workflows. Clinical applications for gel dosimetry, external beam planning, and MRI-ultrasound image fusion in brachytherapy were rapidly prototyped resulting robust applications that are already in use in clinical research. The conversion algorithms were found to be accurate and reliable using these applications.

Conclusions

A generic software library has been designed and developed for automatic management of multiple data formats in segmentation tasks. It enhances both user and developer experience, enabling fast and convenient manual workflows and quicker and more robust software prototyping. The software’s BSD-style open-source license allows complete freedom of use of the library.

Introduction

Segmentation, which is the process of delineating structures of interest, is a critically important task in many medical image computing and visualization workflows: it enables quantitative analysis inside structures such as organs and malignancies, facilitates therapy planning, and provides better understanding of patient anatomy. Various data structures are available to represent segmentation results. Unfortunately none of them are optimal for storage, analysis, and real-time visualization at the same time, so a trade-off is typically made to choose the most suitable representation for the main purpose of a specific application.

Most commonly, segmentation results are stored in 3D binary labelmap volume (shown in Fig. 1/A) where the value of each voxel indicates if that point is inside or outside the structure. This representation is the usual output of manual segmentation and preferred input for image processing and analysis algorithms. However, for 3D visualization of structures a surface model (Fig. 1/B) is more optimal, as it can be rendered by graphics cards efficiently. A surface model is a mesh consisting of tens or hundreds of thousands of connected triangles. Certain segmentation algorithms yield fractional labelmaps (Fig. 1/C) with voxels indicating probabilities [7] instead of a binary decision. For radiation therapy, the DICOM standard [11] requires the structures to be stored as “structure sets” - a series of planar contours (Fig. 1/D). Structure sets may also be represented as ribbons (Fig. 1/E), which can be easily computed from contours, indicate slice thickness, and make 3D shapes more recognizable.

In most workflows, it is not sufficient to use a single representation of segmentation data. An example for the need of multiple representations is the calculation of dose volume histograms (DVH) for radiation therapy, which are used for treatment plan evaluation and optimization. Targets to treat and organs to avoid are segmented by delineating their cross-section in multiple slices of the image, which results in a list of planar contours. Contours are then voxelized into labelmaps, so that they can be used as masks for dose distribution volumes and computing the histograms. For 3D visualization closed surfaces are shown, which can be created either from contours or labelmaps.

There are challenges involved in the design and implementation of storage and analysis of anatomical structures that must be addressed in all software applications:

  • 1.

    Conversion method selection: The need for conversion in order to prepare segmentation data for certain operations is something to be recognized by the user. Further, some details are often lost or altered during conversion. It is important to make users aware of these changes, while at the same time keep the software simple and easy to use.

  • 2.

    Provenance: The identity and origin of the structures and what they represent. In a typical research workflow not driven by a rigid processing pipeline it is up to the user to maintain the objects participating in the workflow. Often, this results in an unorganized set of objects with arbitrary names. When working with more than a few structures it quickly becomes unclear where each object originates from and what transformations they went through.

  • 3.

    Consistency: Representations may change after conversions (e.g. by manual editing), in which case the other representations of the same structure become invalid, and the data scene inconsistent. It is imperative to make sure that no invalid data is accessible at any time.

  • 4.

    Coherence: Structure sets typically correspond to the same entity (i.e., a patient), so when objects are stored in memory or disk, processed, or visualized it should be possible to manage them as a unified whole.

Extensive research has been performed towards developing fast and accurate conversion algorithms between representations ([2], [5]; Jian [8], [10], [13], [18], etc.) for various applications ranging from geography, through computer graphics to medical image computing. To our knowledge, however, no investigation has been done in terms of a complex workflow involving multiple data representations and the challenges identified earlier. Keeping the data in good shape is essential, especially if a human user is involved, or the workflow steps’ order is not fixed. By making these conversion methods more accessible and robust, the time spent in the workflow can be reduced considerably. An example is the dosimetry workflow by Alexander et al. [1], in which the time of analysis was reduced from hours to minutes.

This paper proposes a software system methodology and implementation for Polymorphic Segmentation Representation to facilitate dynamic management of multiple data formats representing the same structure in a way that addresses the identified challenges involved in the process. It aspires to open the way to rapid prototyping of more robust and user-friendly software for medical image analysis research and related clinical applications.

Section snippets

Computational methods and theory

A complex data structure called a segmentation object is proposed that unites different representations of segmented structures. It provides automatic conversion between representation types. New representation types can be dynamically added to the system. The software design is the result of a consensus reached through discussions with numerous researchers and developers participating in the Project Week open-source programming event series [9].

Core library

The basic storage and conversion features are implemented in a software library called PolySeg.1 It includes the segmentation object and segment classes, as well as those responsible for conversion. The core library is written entirely in C++ and is based on the Visualization Toolkit (VTK) [17]. VTK is well-established and one of the most widely used software libraries in the field of medical image computing: MITK [12], MeVisLab [16], medInria [24], NIfTI [3],

Samples of typical system

It has been shown through various applications that building on PolySeg yields robust and versatile software in a short development time. The following applications are based on 3D Slicer, being the first fully supported platform utilizing the PolySeg features. However, similar applications are possible to develop using PolySeg directly, or after integration to a similar medical image computing toolkit.

Hardware and software specifications

PolySeg is platform-independent, and is tested on Windows, MacOS, and Linux. Nightly automated testing ensures error-free operation on each operating system. The end-user application prototypes implemented using PolySeg and 3D Slicer work on the same three operating systems.

Hardware requirements are specific to the end-user applications built on top of PolySeg. However, the general requirement is a configuration that has the graphics capabilities to display and memory to hold the original

Mode of availability of the system and programs

The PolySeg library is distributed under the BSD 2-clause open-source license, which contains no restrictions on the use of the software. The source code and documentation are accessible freely on GitHub: https://github.com/PerkLab/PolySeg.

The implemented end-user applications are also under the BSD 2-clause license, and can be freely accessed at the following locations:

  • Segment Editor within the 3D Slicer application: https://www.slicer.org

  • Batch structure set conversion and open-source external

Acknowledgement

This work was supported by Cancer Care Ontario through the Applied Cancer Research Unit and Research Chair in Cancer Imaging grant, The Ontario Consortium for Adaptive Interventions in Radiation Oncology (OCAIRO) grant funded by the Ontario Research Fund, and the Research Software Program provided by CANARIE.

Conflict of interest statement

The authors have no relevant conflicts of interest to disclose.

References (27)

  • T. Kapur et al.

    Increasing the impact of medical image computing using community-based open- access Hackathons : the NA-MIC and 3D Slicer experience

  • D. Meyers et al.

    Surfaces from contours

    ACM Trans. Graph.

    (1992)
  • P. Mildenberger et al.

    Introduction to the DICOM standard

    Eur. Radiol.

    (2002)
  • Cited by (48)

    • VT3D: a visualization toolbox for 3D transcriptomic data

      2023, Journal of Genetics and Genomics
    View all citing articles on Scopus
    View full text