A Survey of Methods for 3D Histology Reconstruction
Graphical abstract
Introduction
Histology is concerned with the various methods of microscopic examination of a thin tissue section (or slice) (Culling, 2013), most commonly sampled from a specimen post mortem or from a biopsy. Cutting through a specimen reveals its internal topography and staining the sections permits the observation of complex differentiated structures. Then, the digitisation of histological sections (referred to as digital pathology) makes high-resolution microscope sections available for image computing and machine learning algorithms. These allow for disease detection, characterisation and prediction so as to complement the opinion of the pathologist (Madabhushi and Lee, 2016) and constitute the field of histopathological image analysis (Gurcan et al., 2009).
When willing to extend such examinations to the third dimension, one faces the following problem: starting from (a series of) 2D samples, how to regain information of the structure in 3D? Volume slicing breaks the spatial relations between structures and creates discontinuities which hamper mental representations in 3D and thereby, a full understanding of the anatomy. In this respect, Gagnier and Shipley (2013) described the complexity in determining the progression of features into a volume by solely relying on a single face.
In addition, structures are independently altered due to the histological preparation itself (Fig. 1). This may cause anatomically different structures to look similar in microscope slides and conversely, slicing may cause one same structure to have different views if not consistent. Other changes have to do with objects that may disappear or become highly salient from one section to another due to staining variability.
Although humans can represent and mentally transform the shapes of objects very well, this ability worsens when structures are interconnected within a dense and complicated environment, or subject to complex transformations (Atit, Shipley, Tikoff, 2013, Frick, Möhring, Newcombe, 2014). Reconstructing histology volumes from serial 2D sections thus seems natural in order to (re)gain knowledge about spatial environments in 3D, while accessing microscopic information about tissues. In this regard, the Swiss anatomist Wilhelm His Sr. (1831-1904) best explained that “just looking through sections does not enable one to build three-dimensional images in the mind and those who wish to grasp anatomical structures must actively engage in working through a reconstruction, reproducing the relationships they wish to understand” (Hopwood, 1999).
When using histology alone, reconstruction algorithms aim to restore continuity and usually exploit the fact that the biological specimen’s shape changes smoothly across sections. In other words, a set of slices is assumed available, with appropriate spacing (i.e., not too sparse) so that one can define a (reconstructed) volume. Such algorithms provide a representation of structures and their environment in three dimensions, although one needs to bear in mind that the original shape is unattainable without prior knowledge. For illustration purposes, Malandain et al. (2004) pointed out that if a banana is sliced, an ellipsoid will be reconstructed through pairwise alignment of adjacent slices, rather than the original fruit. This is called the “banana effect” or “z-shift”.
The most direct way of recovering volumes from sets of 2D serial histological sections is by optimising the spatial alignment of every pair of adjacent images using registration techniques. Image registration permits the automation of this transformation process, and allows to redefine "visual closeness" as the optimisation of a certain cost function. It also accounts for the complex transformations that affect hitological sections individually and grants higher reproducibility with less or no human effort. Composing the transformations from every image to a reference image completes the process—the reference section being chosen for its high contrast, its small amount of artefacts, and preferably but not necessarily its location in the middle of the stack. A consequence is that any registration error impacts the final reconstruction by propagation due to the sequential nature of the procedure. Methods have therefore been developed to minimise these effects by looking at neighbourhoods rather than single slices in order to smooth those errors out; attention has also been directed toward preprocessing the images of tissue sections owing to their highly variable quality.
A remedy to the incorrectness of the histology reconstruction is the use of 3D medical images, such as magnetic resonance imaging (MRI). By providing structural ground truth, they refine the space of solutions—although registration itself remains an ill-posed problem. Careful use of registration techniques can produce histology reconstructions closer to reality, establish more accurate correspondences across modalities and thereby contribute to more sound data analyses. Two cases are commonly encountered: (i) only a single (or too few) histological sections are available (like for biopsies), whereby a volume reconstruction is meaningless and one solely aims to identify the corresponding (resampled) MRI plane in order to deform histology correctly. In that case, one cannot rely on the greater supports that volumes offer, and such a situation calls for careful initialisation and 2D-3D registration methods (Ferrante and Paragios, 2017); (ii) a sufficient number of histological sections is available (i.e., the set spans several MRI slices) and one can thus manipulate volumes, globally bring them into spatial alignment, and non-linearly register each slice with its corresponding (resampled) MRI plane. In the process of relating in vivo to post mortem, it is not uncommon to use intermediate modalities (Fig. 2), such as blockface photographs (pictures of the tissue face taken prior to cutting), so as to keep track on the deformations that the tissue undergoes during its changes of state; or take advantage of needles, which allow for straightforward extraction and matching of landmarks in both modalities.
Besides providing structural ground truth, 3D medical imaging constitutes an invaluable source for accurate, non-invasive study of biological structures and their functions. Relative to histology, Fischl (2013) listed three advantages: the possibility of (i) imaging the exact same tissue with multiple contrasts (e.g., T1 or T2w MRI, MTR, etc.); (ii) imaging large samples (e.g., whole-brain or whole-hemisphere) with much less effort than e.g., whole-brain or prostate whole-mount histology; (iii) preserving the geometry of the sample and avoiding irreversible damage and distortions induced by processing, cutting, mounting and staining during the histological preparation.
With respect to resolution, MR imaging is outperformed by histology (< 1 µm). In addition, for many pathological disorders, there is still no no sequence acquisition that allows imaging to be a full substitute for histology. This is due to the poorly understood relationships between histological and magnetic properties of tissues. Directly predicting the imaging appearance of a histological signature is therefore extremely complex. Practically, this results in that different pathologies can share a common imaging phenotype (Gore, 2015). For example, Filippi et al. (2012) noted that in proton density, FLAIR and T2w MRI scans, Multiple Sclerosis (MS) lesions appeared as non-specific focal areas of signal increase and, therefore, resembled many other types of pathology. This makes it difficult to differentiate them with imaging only. Additionally, some cortical MS lesions can still be missed with conventional MRI sequences (Seewann et al., 2012). Direct comparisons with histology helps interpret images better and derive more information. They may also help in correcting or adjusting existing imaging protocols in order to optimally visualise pathological markers (e.g., lesions in the grey matter of patients with MS).
One of the many benefits of combining histology and medical imaging is to confirm non-invasive measures with baseline information on the actual properties of tissues (Annese, 2012). By combining 3D medical imaging with digital pathology, it is possible to simultaneously obtain the rich structural information of the former and the chemical and cellular information of the latter, which may allow for more complete characterisation and understanding of e.g., diseases (Mori, 2016). One can also derive more accurate segmentations of architectonic boundaries to be used in the creation of atlases (Ding, Royall, Sunkin, Ng, Facer, Lesnar, Guillozet-Bongaarts, McMurray, Szafer, Dolbeare, et al., 2016, Oh, Harris, Ng, Winslow, Cain, Mihalas, Wang, Lau, Kuan, Henry, et al., 2014, Amunts, Lepage, Borgeat, Mohlberg, Dickscheid, Rousseau, Bludau, Bazin, Lewis, Oros-Peusquens, et al., 2013, Hawrylycz, Lein, Guillozet-Bongaarts, Shen, Ng, Miller, Van De Lagemaat, Smith, Ebbert, Riley, et al., 2012, Yushkevich, Avants, Pluta, Das, Minkoff, Mechanic-Hamilton, Glynn, Pickup, Liu, Gee, et al., 2009) as well as brain mapping (Amunts and Zilles, 2015). Such undertakings are intended to eventually bridge the gap between in vivo and post mortem studies.
Currently, direct overall visual comparison is considered acceptable to assess the correlation between histopathological findings and imaging observations. On that matter though, it was recently mentioned in the context of prostate cancer assessment that due to variations in imaging technologies, contouring procedures and data analyses, available volume correlation studies had yielded conflicting results (Priester et al., 2016). Such contradictions were explained by the worrying observation that nearly all prior attempts to define MRI/pathological relationships had relied on imprecise techniques such as manual registration, volume approximation, and 2D measurements. Following the same line of thought—two decades before—correlation was proved to be optimised when the alignment between data had first been carefully taken care of by use of a combination of linear and non-linear transformations (Mazziotta et al., 1995). In other words, ensuring the comparison of like with like is of utmost importance (Madabhushi and Lee, 2016). In this paper, we describe methodologies which relied on (automatic) image registration techniques.
The objective of this paper is to survey the past 30 years of literature on 3D histology reconstruction. The paper is structured according to the multidisciplinary nature of the problem. Sections 2 and 3 explain the preparation of histological slices, list artefacts associated with every step of the process and cover preprocessing methods in order to best cope with image deteriorations. Section 4 proposes a classification of methods for 3D histology reconstruction from 2D serial sections and Section 5 describes pipelines that aim to combine histological and clinical imaging information. Section 6 presents approaches used to validate the correctness of reconstructions—with or without the help of external information—and Section 7 enumerates the clinical applications of such techniques. Finally, Section 8 returns on a few methodological points, discusses some of the remaining challenges in the field and highlights the importance of cross-disciplinary knowledge in solving a biological question.
Section snippets
From fresh tissue to digital pathology
A pathologist receiving fresh tissue has three options: keeping it fresh, stabilising it in a fixative, or freezing it. Biological tissue is too soft for direct sectioning (although a vibrating blade might work), so it is most commonly either embedded in a hardening material and sectioned using a microtome, or frozen and sectioned in a cryostat (a microtome inside a freezer). Sections are then mounted on glass slides and stained before being observed under the microscope by the
Preprocessing of digital pathology
Among the artefacts resulting from histological preparation, loss of detail and changes in morphology burden image analysis. Not much can be done about them as content is hardly retrievable from lost or corrupted information without any prior knowledge. When due to scanning, though (local poor focusing can cause blurred regions in images), loss of detail is surmountable but at the cost of time-consuming review by the scanner operator. In the context of whole slide imaging, Lopez et al. (2013)
3D histology reconstruction
Histology reconstruction methods aim to restore the loss of continuity due to volume slicing. They are based on the assumption that the shape of a biological specimen changes smoothly across sections, but suffer from the various artefacts that affect every section independently during preparation.
When using histology alone, reconstruction algorithms provide representations of structures and their environment in 3D—which help with subsequent segmentation and classification tasks (McCann et al.,
Histology reconstruction using 3D medical images
This section presents pipelines that aim to improve histology reconstructions with the help of 3D medical images. As mentioned earlier, this supposes the access to a suitable set of histological slices. By suitable we mean that a sufficient number of sections with an appropriate spacing between them (relative to the MRI slices thickness) is available. Hence the slight abuse of language made in the section (and the paper) title in cases where only a single or too few histological slices are
Validation methods
We hereafter detail the ways authors have validated the accuracy and the precision of image registration, as defined by Maintz and Viergever (1998), in the context of histology reconstruction (with or without the help of medical imaging).
Applications
We underline three main areas of applications within which the covered literature falls into: (i) examining structures with respect to their environment in 3D (Section 7.1) with or without the help of 3D medical imaging; (ii) the correlation of data (Section 7.2), which benefits from the access to the underlying microbiology to improve the characterisation/discrimination of signals in non-invasive imaging; and (iii) the creation of digital atlases (Section 7.3), which allows for easy 2D and 3D
Discussion and perspectives
This section covers three topics: (Section 8.1) some methodological comments on pipelines, their differences, advantages and drawbacks; (Section 8.2) some of the remaining challenges; and (Section 8.3) concluding remarks on the importance of cross-disciplinary knowledge in solving the biological question associated with histology-MRI registration. Note that the discussion is directed towards the multimodal correspondence problem that underpins 3D histology reconstruction with the help of
Acknowledgements
The authors would like to thank Dr. Smriti Patodia, from UCL Institute of Neurology (Department of Neuropathology), for her comments on Section 2 and the images used in Figures 2 and 5, as well as Dr. Steven Van de Pavert, Mr. Marcello Moccia, and Prof. Olga Ciccarelli from UCL Institute of Neurology (NMR Research Unit), for the images used in Figure 6.
This research was supported by the European Research Council (Starting Grant 677697, project BUNGEE-TOOLS), the EPSRC (EP/H046410/1,
References (423)
- et al.
Histology-derived volumetric annotation of the human hippocampal subfields in postmortem mri
Neuroimage
(2014) - et al.
Multi-atlas based segmentation of brain images: atlas selection and its effect on accuracy
Neuroimage
(2009) - et al.
Architectonic mapping of the human brain beyond brodmann
Neuron
(2015) - et al.
Computer-assisted alignment of standard serial sections without use of artificial landmarks. a practical approach to the utilization of incomplete information in 3-d reconstruction of the hippocampal region
Journal of neuroscience methods
(1992) - et al.
Free-d: an integrated environment for three-dimensional reconstruction from serial sections
Journal of neuroscience methods
(2005) - et al.
Postmortem high-resolution 3-dimensional imaging of the primate brain: blockface imaging of perfusion stained tissue
Neuroimage
(2006) - et al.
Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain
Medical image analysis
(2008) - et al.
A novel approach to the human connectome: ultra-high resolution mapping of fiber tracts in the brain
Neuroimage
(2011) - et al.
Color standardization in whole slide imaging using a color calibration slide
Journal of pathology informatics
(2014) - et al.
Improving the visualization and detection of tissue folds in whole slide images through color enhancement
Journal of pathology informatics
(2010)
An assessment of methods for aligning two-dimensional microscope sections to create image volumes
Journal of neuroscience methods
A robust and extendible framework for medical image registration focused on rapid clinical application deployment
Computers in biology and medicine
Correcting spatial distortion in histological images
Computerized Medical Imaging and Graphics
A technique for quantitative three-dimensional analysis of microvascular structure
Microvascular research
Histology safety: now and then
Annals of diagnostic pathology
Mapping of histologically identified long fiber tracts in human cerebral hemispheres to the mri volume of a reference brain: position and spatial variability of the optic radiation
Neuroimage
Transformation diffusion reconstruction of three-dimensional histology volumes from two-dimensional image stacks
Medical image analysis
The creation of a brain atlas for image guided neurosurgery using serial histological data
Neuroimage
Histostitcher©: An interactive program for accurate and rapid reconstruction of digitized whole histological sections from tissue fragments
Computerized Medical Imaging and Graphics
Semi-automated reconstruction of neural circuits using electron microscopy
Current opinion in neurobiology
Accuracy of image registration between mri and light microscopy in the ex vivo brain
Magnetic resonance imaging
Smoothness-guided 3-d reconstruction of 2-d histological images
Neuroimage
Brainvisa: software platform for visualization and analysis of multi-modality brain data
Neuroimage
Feature-based registration of histopathology images with different stains: an application for computerized follicular lymphoma prognosis
Computer methods and programs in biomedicine
Three-dimensional reconstruction of stained histological slices and 3d non-linear registration with in-vivo mri for whole baboon brain
Journal of neuroscience methods
Comparison of fiber tracts derived from in-vivo dti tractography with 3d histological neural tract tracer reconstruction on a macaque brain
Neuroimage
Postmortem magnetic resonance imaging to guide the pathologic cut
Journal of Neuropathology & Experimental Neurology
Statistical distributions in the folding of elastic structures
Journal of Statistical Mechanics: Theory and Experiment
Probabilistic atlas of the human hippocampus combining ex vivo mri and histology
International Conference on Medical Image Computing and Computer-Assisted Intervention
Kaze features
European Conference on Computer Vision
Fast explicit diffusion for accelerated features in nonlinear scale spaces
British Machine Vision Conference (BMVC)
Multimodal whole brain registration: Mri and high resolution histology
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops
Registering coronal histological 2-d sections of a rat brain with coronal sections of a 3-d brain atlas using geometric curve invariants and b-spline representation
IEEE Transactions on Medical Imaging
Facilitating tumor functional assessment by spatially relating 3d tumor histology and in vivo mri: image registration approach
PLoS One
The principal axes transformation-a method for image registration
J Nucl Med
Bigbrain: an ultrahigh-resolution 3d human brain model
Science
Imaging mass spectrometry of proteins and peptides: 3d volume reconstruction
Nature Methods
The importance of combining mri and large-scale digital histology in neuroimaging studies of brain connectivity
Mapping the connectome: Multi-level analysis of brain connectivity
Postmortem examination of patient hms brain based on histological sectioning and digital 3d reconstruction
Nature communications
3d reconstruction of histological sections: Application to mammary gland tissue
Microscopy research and technique
bunwarpj: Consistent and elastic registration in imagej, methods and applications
Second ImageJ User & Developer Conference
Twisting space: are rigid and non-rigid mental transformations separate spatial skills?
Cognitive processing
Some informational aspects of visual perception
Psychological review
An automatic nonrigid registration for stained histological sections
IEEE Transactions on Image Processing
Advanced normalization tools (ants)
Insight J
An open source multivariate framework for n-tissue segmentation with evaluation on public data
Neuroinformatics
Consistency and standardization of color in medical imaging: a consensus report
Journal of digital imaging
Automatic best reference slice selection for smooth volume reconstruction of a mouse brain from histological images
IEEE Transactions on Medical imaging
Registration of serial sections of mouse liver cell nuclei
Journal of microscopy
Three-dimensional volume reconstruction of extracellular matrix proteins in uveal melanoma from fluorescent confocal laser scanning microscope images
Journal of microscopy
Cited by (136)
Joint 2D to 3D image registration workflow for comparing multiple slice photographs and CT scans of apple fruit with internal disorders
2024, Postharvest Biology and TechnologyA 3-dimensional histology computer model of malignant melanoma and its implications for digital pathology
2023, European Journal of CancerWide field block face imaging using deep ultraviolet induced autofluorescence of the human brain
2023, Journal of Neuroscience MethodsRetrieval of siltation 3D properties in artificially created water conveyance tunnels using image-based 3D reconstruction
2023, Measurement: Journal of the International Measurement ConfederationDeep feature based cross-slide registration
2023, Computerized Medical Imaging and Graphics