Elsevier

Medical Image Analysis

Volume 46, May 2018, Pages 73-105
Medical Image Analysis

A Survey of Methods for 3D Histology Reconstruction

https://doi.org/10.1016/j.media.2018.02.004Get rights and content

Highlights

  • We describe the process of generating histological sections.

  • We present artefacts and image processing methods to minimise them.

  • We survey methods for 3D histology reconstruction.

  • We highlight hybrid approaches and discuss remaining challenges in the field.

Abstract

Histology permits the observation of otherwise invisible structures of the internal topography of a specimen. Although it enables the investigation of tissues at a cellular level, it is invasive and breaks topology due to cutting. Three-dimensional (3D) reconstruction was thus introduced to overcome the limitations of single-section studies in a dimensional scope. 3D reconstruction finds its roots in embryology, where it enabled the visualisation of spatial relationships of developing systems and organs, and extended to biomedicine, where the observation of individual, stained sections provided only partial understanding of normal and abnormal tissues. However, despite bringing visual awareness, recovering realistic reconstructions is elusive without prior knowledge about the tissue shape.

3D medical imaging made such structural ground truths available. In addition, combining non-invasive imaging with histology unveiled invaluable opportunities to relate macroscopic information to the underlying microscopic properties of tissues through the establishment of spatial correspondences; image registration is one technique that permits the automation of such a process and we describe reconstruction methods that rely on it. It is thereby possible to recover the original topology of histology and lost relationships, gain insight into what affects the signals used to construct medical images (and characterise them), or build high resolution anatomical atlases.

This paper reviews almost three decades of methods for 3D histology reconstruction from serial sections, used in the study of many different types of tissue. We first summarise the process that produces digitised sections from a tissue specimen in order to understand the peculiarity of the data, the associated artefacts and some possible ways to minimise them. We then describe methods for 3D histology reconstruction with and without the help of 3D medical imaging, along with methods of validation and some applications. We finally attempt to identify the trends and challenges that the field is facing, many of which are derived from the cross-disciplinary nature of the problem as it involves the collaboration between physicists, histolopathologists, computer scientists and physicians.

Introduction

Histology is concerned with the various methods of microscopic examination of a thin tissue section (or slice) (Culling, 2013), most commonly sampled from a specimen post mortem or from a biopsy. Cutting through a specimen reveals its internal topography and staining the sections permits the observation of complex differentiated structures. Then, the digitisation of histological sections (referred to as digital pathology) makes high-resolution microscope sections available for image computing and machine learning algorithms. These allow for disease detection, characterisation and prediction so as to complement the opinion of the pathologist (Madabhushi and Lee, 2016) and constitute the field of histopathological image analysis (Gurcan et al., 2009).

When willing to extend such examinations to the third dimension, one faces the following problem: starting from (a series of) 2D samples, how to regain information of the structure in 3D? Volume slicing breaks the spatial relations between structures and creates discontinuities which hamper mental representations in 3D and thereby, a full understanding of the anatomy. In this respect, Gagnier and Shipley (2013) described the complexity in determining the progression of features into a volume by solely relying on a single face.

In addition, structures are independently altered due to the histological preparation itself (Fig. 1). This may cause anatomically different structures to look similar in microscope slides and conversely, slicing may cause one same structure to have different views if not consistent. Other changes have to do with objects that may disappear or become highly salient from one section to another due to staining variability.

Although humans can represent and mentally transform the shapes of objects very well, this ability worsens when structures are interconnected within a dense and complicated environment, or subject to complex transformations (Atit, Shipley, Tikoff, 2013, Frick, Möhring, Newcombe, 2014). Reconstructing histology volumes from serial 2D sections thus seems natural in order to (re)gain knowledge about spatial environments in 3D, while accessing microscopic information about tissues. In this regard, the Swiss anatomist Wilhelm His Sr. (1831-1904) best explained that “just looking through sections does not enable one to build three-dimensional images in the mind and those who wish to grasp anatomical structures must actively engage in working through a reconstruction, reproducing the relationships they wish to understand” (Hopwood, 1999).

When using histology alone, reconstruction algorithms aim to restore continuity and usually exploit the fact that the biological specimen’s shape changes smoothly across sections. In other words, a set of slices is assumed available, with appropriate spacing (i.e., not too sparse) so that one can define a (reconstructed) volume. Such algorithms provide a representation of structures and their environment in three dimensions, although one needs to bear in mind that the original shape is unattainable without prior knowledge. For illustration purposes, Malandain et al. (2004) pointed out that if a banana is sliced, an ellipsoid will be reconstructed through pairwise alignment of adjacent slices, rather than the original fruit. This is called the “banana effect” or “z-shift”.

The most direct way of recovering volumes from sets of 2D serial histological sections is by optimising the spatial alignment of every pair of adjacent images using registration techniques. Image registration permits the automation of this transformation process, and allows to redefine "visual closeness" as the optimisation of a certain cost function. It also accounts for the complex transformations that affect hitological sections individually and grants higher reproducibility with less or no human effort. Composing the transformations from every image to a reference image completes the process—the reference section being chosen for its high contrast, its small amount of artefacts, and preferably but not necessarily its location in the middle of the stack. A consequence is that any registration error impacts the final reconstruction by propagation due to the sequential nature of the procedure. Methods have therefore been developed to minimise these effects by looking at neighbourhoods rather than single slices in order to smooth those errors out; attention has also been directed toward preprocessing the images of tissue sections owing to their highly variable quality.

A remedy to the incorrectness of the histology reconstruction is the use of 3D medical images, such as magnetic resonance imaging (MRI). By providing structural ground truth, they refine the space of solutions—although registration itself remains an ill-posed problem. Careful use of registration techniques can produce histology reconstructions closer to reality, establish more accurate correspondences across modalities and thereby contribute to more sound data analyses. Two cases are commonly encountered: (i) only a single (or too few) histological sections are available (like for biopsies), whereby a volume reconstruction is meaningless and one solely aims to identify the corresponding (resampled) MRI plane in order to deform histology correctly. In that case, one cannot rely on the greater supports that volumes offer, and such a situation calls for careful initialisation and 2D-3D registration methods (Ferrante and Paragios, 2017); (ii) a sufficient number of histological sections is available (i.e., the set spans several MRI slices) and one can thus manipulate volumes, globally bring them into spatial alignment, and non-linearly register each slice with its corresponding (resampled) MRI plane. In the process of relating in vivo to post mortem, it is not uncommon to use intermediate modalities (Fig. 2), such as blockface photographs (pictures of the tissue face taken prior to cutting), so as to keep track on the deformations that the tissue undergoes during its changes of state; or take advantage of needles, which allow for straightforward extraction and matching of landmarks in both modalities.

Besides providing structural ground truth, 3D medical imaging constitutes an invaluable source for accurate, non-invasive study of biological structures and their functions. Relative to histology, Fischl (2013) listed three advantages: the possibility of (i) imaging the exact same tissue with multiple contrasts (e.g., T1 or T2w MRI, MTR, etc.); (ii) imaging large samples (e.g., whole-brain or whole-hemisphere) with much less effort than e.g., whole-brain or prostate whole-mount histology; (iii) preserving the geometry of the sample and avoiding irreversible damage and distortions induced by processing, cutting, mounting and staining during the histological preparation.

With respect to resolution, MR imaging is outperformed by histology (< 1 µm). In addition, for many pathological disorders, there is still no no sequence acquisition that allows imaging to be a full substitute for histology. This is due to the poorly understood relationships between histological and magnetic properties of tissues. Directly predicting the imaging appearance of a histological signature is therefore extremely complex. Practically, this results in that different pathologies can share a common imaging phenotype (Gore, 2015). For example, Filippi et al. (2012) noted that in proton density, FLAIR and T2w MRI scans, Multiple Sclerosis (MS) lesions appeared as non-specific focal areas of signal increase and, therefore, resembled many other types of pathology. This makes it difficult to differentiate them with imaging only. Additionally, some cortical MS lesions can still be missed with conventional MRI sequences (Seewann et al., 2012). Direct comparisons with histology helps interpret images better and derive more information. They may also help in correcting or adjusting existing imaging protocols in order to optimally visualise pathological markers (e.g., lesions in the grey matter of patients with MS).

One of the many benefits of combining histology and medical imaging is to confirm non-invasive measures with baseline information on the actual properties of tissues (Annese, 2012). By combining 3D medical imaging with digital pathology, it is possible to simultaneously obtain the rich structural information of the former and the chemical and cellular information of the latter, which may allow for more complete characterisation and understanding of e.g., diseases (Mori, 2016). One can also derive more accurate segmentations of architectonic boundaries to be used in the creation of atlases (Ding, Royall, Sunkin, Ng, Facer, Lesnar, Guillozet-Bongaarts, McMurray, Szafer, Dolbeare, et al., 2016, Oh, Harris, Ng, Winslow, Cain, Mihalas, Wang, Lau, Kuan, Henry, et al., 2014, Amunts, Lepage, Borgeat, Mohlberg, Dickscheid, Rousseau, Bludau, Bazin, Lewis, Oros-Peusquens, et al., 2013, Hawrylycz, Lein, Guillozet-Bongaarts, Shen, Ng, Miller, Van De Lagemaat, Smith, Ebbert, Riley, et al., 2012, Yushkevich, Avants, Pluta, Das, Minkoff, Mechanic-Hamilton, Glynn, Pickup, Liu, Gee, et al., 2009) as well as brain mapping (Amunts and Zilles, 2015). Such undertakings are intended to eventually bridge the gap between in vivo and post mortem studies.

Currently, direct overall visual comparison is considered acceptable to assess the correlation between histopathological findings and imaging observations. On that matter though, it was recently mentioned in the context of prostate cancer assessment that due to variations in imaging technologies, contouring procedures and data analyses, available volume correlation studies had yielded conflicting results (Priester et al., 2016). Such contradictions were explained by the worrying observation that nearly all prior attempts to define MRI/pathological relationships had relied on imprecise techniques such as manual registration, volume approximation, and 2D measurements. Following the same line of thought—two decades before—correlation was proved to be optimised when the alignment between data had first been carefully taken care of by use of a combination of linear and non-linear transformations (Mazziotta et al., 1995). In other words, ensuring the comparison of like with like is of utmost importance (Madabhushi and Lee, 2016). In this paper, we describe methodologies which relied on (automatic) image registration techniques.

The objective of this paper is to survey the past 30 years of literature on 3D histology reconstruction. The paper is structured according to the multidisciplinary nature of the problem. Sections 2 and 3 explain the preparation of histological slices, list artefacts associated with every step of the process and cover preprocessing methods in order to best cope with image deteriorations. Section 4 proposes a classification of methods for 3D histology reconstruction from 2D serial sections and Section 5 describes pipelines that aim to combine histological and clinical imaging information. Section 6 presents approaches used to validate the correctness of reconstructions—with or without the help of external information—and Section 7 enumerates the clinical applications of such techniques. Finally, Section 8 returns on a few methodological points, discusses some of the remaining challenges in the field and highlights the importance of cross-disciplinary knowledge in solving a biological question.

Section snippets

From fresh tissue to digital pathology

A pathologist receiving fresh tissue has three options: keeping it fresh, stabilising it in a fixative, or freezing it. Biological tissue is too soft for direct sectioning (although a vibrating blade might work), so it is most commonly either embedded in a hardening material and sectioned using a microtome, or frozen and sectioned in a cryostat (a microtome inside a freezer). Sections are then mounted on glass slides and stained before being observed under the microscope by the

Preprocessing of digital pathology

Among the artefacts resulting from histological preparation, loss of detail and changes in morphology burden image analysis. Not much can be done about them as content is hardly retrievable from lost or corrupted information without any prior knowledge. When due to scanning, though (local poor focusing can cause blurred regions in images), loss of detail is surmountable but at the cost of time-consuming review by the scanner operator. In the context of whole slide imaging, Lopez et al. (2013)

3D histology reconstruction

Histology reconstruction methods aim to restore the loss of continuity due to volume slicing. They are based on the assumption that the shape of a biological specimen changes smoothly across sections, but suffer from the various artefacts that affect every section independently during preparation.

When using histology alone, reconstruction algorithms provide representations of structures and their environment in 3D—which help with subsequent segmentation and classification tasks (McCann et al.,

Histology reconstruction using 3D medical images

This section presents pipelines that aim to improve histology reconstructions with the help of 3D medical images. As mentioned earlier, this supposes the access to a suitable set of histological slices. By suitable we mean that a sufficient number of sections with an appropriate spacing between them (relative to the MRI slices thickness) is available. Hence the slight abuse of language made in the section (and the paper) title in cases where only a single or too few histological slices are

Validation methods

We hereafter detail the ways authors have validated the accuracy and the precision of image registration, as defined by Maintz and Viergever (1998), in the context of histology reconstruction (with or without the help of medical imaging).

Applications

We underline three main areas of applications within which the covered literature falls into: (i) examining structures with respect to their environment in 3D (Section 7.1) with or without the help of 3D medical imaging; (ii) the correlation of data (Section 7.2), which benefits from the access to the underlying microbiology to improve the characterisation/discrimination of signals in non-invasive imaging; and (iii) the creation of digital atlases (Section 7.3), which allows for easy 2D and 3D

Discussion and perspectives

This section covers three topics: (Section 8.1) some methodological comments on pipelines, their differences, advantages and drawbacks; (Section 8.2) some of the remaining challenges; and (Section 8.3) concluding remarks on the importance of cross-disciplinary knowledge in solving the biological question associated with histology-MRI registration. Note that the discussion is directed towards the multimodal correspondence problem that underpins 3D histology reconstruction with the help of

Acknowledgements

The authors would like to thank Dr. Smriti Patodia, from UCL Institute of Neurology (Department of Neuropathology), for her comments on Section 2 and the images used in Figures 2 and 5, as well as Dr. Steven Van de Pavert, Mr. Marcello Moccia, and Prof. Olga Ciccarelli from UCL Institute of Neurology (NMR Research Unit), for the images used in Figure 6.

This research was supported by the European Research Council (Starting Grant 677697, project BUNGEE-TOOLS), the EPSRC (EP/H046410/1,

References (423)

  • R. Beare et al.

    An assessment of methods for aligning two-dimensional microscope sections to create image volumes

    Journal of neuroscience methods

    (2008)
  • T. Boehler et al.

    A robust and extendible framework for medical image registration focused on rapid clinical application deployment

    Computers in biology and medicine

    (2011)
  • M.S. Breen et al.

    Correcting spatial distortion in histological images

    Computerized Medical Imaging and Graphics

    (2005)
  • E.M. Brey et al.

    A technique for quantitative three-dimensional analysis of microvascular structure

    Microvascular research

    (2002)
  • R.J. Buesa

    Histology safety: now and then

    Annals of diagnostic pathology

    (2007)
  • U. Bürgel et al.

    Mapping of histologically identified long fiber tracts in human cerebral hemispheres to the mri volume of a reference brain: position and spatial variability of the optic radiation

    Neuroimage

    (1999)
  • R. Casero et al.

    Transformation diffusion reconstruction of three-dimensional histology volumes from two-dimensional image stacks

    Medical image analysis

    (2017)
  • M.M. Chakravarty et al.

    The creation of a brain atlas for image guided neurosurgery using serial histological data

    Neuroimage

    (2006)
  • J. Chappelow et al.

    Histostitcher©: An interactive program for accurate and rapid reconstruction of digitized whole histological sections from tissue fragments

    Computerized Medical Imaging and Graphics

    (2011)
  • D.B. Chklovskii et al.

    Semi-automated reconstruction of neural circuits using electron microscopy

    Current opinion in neurobiology

    (2010)
  • A.S. Choe et al.

    Accuracy of image registration between mri and light microscopy in the ex vivo brain

    Magnetic resonance imaging

    (2011)
  • A. Cifor et al.

    Smoothness-guided 3-d reconstruction of 2-d histological images

    Neuroimage

    (2011)
  • Y. Cointepas et al.

    Brainvisa: software platform for visualization and analysis of multi-modality brain data

    Neuroimage

    (2001)
  • L. Cooper et al.

    Feature-based registration of histopathology images with different stains: an application for computerized follicular lymphoma prognosis

    Computer methods and programs in biomedicine

    (2009)
  • J. Dauguet et al.

    Three-dimensional reconstruction of stained histological slices and 3d non-linear registration with in-vivo mri for whole baboon brain

    Journal of neuroscience methods

    (2007)
  • J. Dauguet et al.

    Comparison of fiber tracts derived from in-vivo dti tractography with 3d histological neural tract tracer reconstruction on a macaque brain

    Neuroimage

    (2007)
  • M. Absinta et al.

    Postmortem magnetic resonance imaging to guide the pathologic cut

    Journal of Neuropathology & Experimental Neurology

    (2014)
  • M. Adda-Bedia et al.

    Statistical distributions in the folding of elastic structures

    Journal of Statistical Mechanics: Theory and Experiment

    (2010)
  • D.H. Adler et al.

    Probabilistic atlas of the human hippocampus combining ex vivo mri and histology

    International Conference on Medical Image Computing and Computer-Assisted Intervention

    (2016)
  • P.F. Alcantarilla et al.

    Kaze features

    European Conference on Computer Vision

    (2012)
  • P.F. Alcantarilla et al.

    Fast explicit diffusion for accelerated features in nonlinear scale spaces

    British Machine Vision Conference (BMVC)

    (2013)
  • M. Alegro et al.

    Multimodal whole brain registration: Mri and high resolution histology

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops

    (2016)
  • W.S.I. Ali et al.

    Registering coronal histological 2-d sections of a rat brain with coronal sections of a 3-d brain atlas using geometric curve invariants and b-spline representation

    IEEE Transactions on Medical Imaging

    (1998)
  • L. Alic et al.

    Facilitating tumor functional assessment by spatially relating 3d tumor histology and in vivo mri: image registration approach

    PLoS One

    (2011)
  • N. Alpert et al.

    The principal axes transformation-a method for image registration

    J Nucl Med

    (1990)
  • K. Amunts et al.

    Bigbrain: an ultrahigh-resolution 3d human brain model

    Science

    (2013)
  • M. Andersson et al.

    Imaging mass spectrometry of proteins and peptides: 3d volume reconstruction

    Nature Methods

    (2008)
  • J. Annese

    The importance of combining mri and large-scale digital histology in neuroimaging studies of brain connectivity

    Mapping the connectome: Multi-level analysis of brain connectivity

    (2012)
  • J. Annese et al.

    Postmortem examination of patient hms brain based on histological sectioning and digital 3d reconstruction

    Nature communications

    (2014)
  • I. Arganda-Carreras et al.

    3d reconstruction of histological sections: Application to mammary gland tissue

    Microscopy research and technique

    (2010)
  • I. Arganda-Carreras et al.

    bunwarpj: Consistent and elastic registration in imagej, methods and applications

    Second ImageJ User & Developer Conference

    (2008)
  • K. Atit et al.

    Twisting space: are rigid and non-rigid mental transformations separate spatial skills?

    Cognitive processing

    (2013)
  • F. Attneave

    Some informational aspects of visual perception

    Psychological review

    (1954)
  • M. Auer et al.

    An automatic nonrigid registration for stained histological sections

    IEEE Transactions on Image Processing

    (2005)
  • B.B. Avants et al.

    Advanced normalization tools (ants)

    Insight J

    (2009)
  • B.B. Avants et al.

    An open source multivariate framework for n-tissue segmentation with evaluation on public data

    Neuroinformatics

    (2011)
  • A. Badano et al.

    Consistency and standardization of color in medical imaging: a consensus report

    Journal of digital imaging

    (2015)
  • U. Bagci et al.

    Automatic best reference slice selection for smooth volume reconstruction of a mouse brain from histological images

    IEEE Transactions on Medical imaging

    (2010)
  • S. Baheerathan et al.

    Registration of serial sections of mouse liver cell nuclei

    Journal of microscopy

    (1998)
  • P. Bajcsy et al.

    Three-dimensional volume reconstruction of extracellular matrix proteins in uveal melanoma from fluorescent confocal laser scanning microscope images

    Journal of microscopy

    (2006)
  • Cited by (136)

    • Deep feature based cross-slide registration

      2023, Computerized Medical Imaging and Graphics
    View all citing articles on Scopus
    View full text