2.1. TomoPy

TomoPy is an open-source Python toolbox to perform tomographic data processing and image reconstruction tasks, developed at the Advanced Photon Source of Argonne National Laboratory (Gürsoy et al., 2014). The toolbox is available for Linux and OS X operating systems, and is aimed at providing a high-level interface for processing and tomographic reconstruction of datasets at synchrotron light sources. TomoPy relies on standard scientific packages like NumPy, SciPy and Scikit, and offers a free, open-source, modular, readable and manageable framework that researchers can use and contribute to easily. Python also offers easy integration with C or Fortran code through shared libraries in situations where computation speed is critical. In addition, the native control software running at several synchrotron facilities, EPICS (https://www.aps.anl.gov/epics), is accessible via Python (https://pyepics.github.io/pyepics/), allowing simultaneous data analysis and real-time feedback on the instrumentation status. So far, TomoPy has been employed in reconstructions for various techniques from micro-CT (Duke et al., 2015) to X-ray fluorescence tomography (Gürsoy et al., 2015a), X-ray scattering tomography (Gürsoy et al., 2015b), Lorentz electron microscopy (Phatak & Gürsoy, 2015) and deployed on large-scale computing facilities (Biçer et al., 2015).

TomoPy includes a plethora of processing functions from pre-processing to image reconstruction of synchrotron tomographic data. It includes ring removal algorithms, such as the generalized Titarenko's algorithm (Miqueles et al., 2014) and a Fourier wavelet approach (Münch et al., 2009), and a zinger correction algorithm based on median filters. The estimation of the rotation center can be calculated using the image entropy calculation based method (Donath et al., 2006) or Vo's Fourier method (Vo et al., 2014). A single-step X-ray phase-retrieval algorithm based on Paganin filtering is available for phase-contrast datasets (Paganin et al., 2002). TomoPy also includes algorithms for post-processing reconstructed images, such as Gaussian filtering and median filtering to reduce noise artifacts.

In addition to Gridrec (Dowd et al., 1999), which is the traditionally used analytical image reconstruction algorithm, TomoPy also offers variants of algebraic reconstruction methods (ART, BART, SIRT) (Kak & Slaney, 2001), and maximum-likelihood expectation maximization (ML-EM) approaches (Dempster et al., 1977), as well as their regularized variations (PML) (Chang et al., 2004). Ordered-subset implementation of all algorithms are also available for efficient calculations; for example, the well known ordered-subset expectation maximization (OSEM) algorithm (Hudson & Larkin, 1994). An overview of all algorithms included in TomoPy, along with their parameters and usage examples, can be found on the documentation website of TomoPy (https://tomopy.readthedocs.org).

Another important property of TomoPy is that it provides X-ray matter interaction simulation tools, such as X-ray transmission or wave propagation, that can be used to evaluate the efficiency of various coding scenarios or as a platform for modeling. The standard installation package of TomoPy is optimized for use at a workstation, but TomoPy algorithms are also suitable for grid-computing and massive parallelization when needed. Experiments with MPI implementations of iterative algorithms and tomography datasets with 1K projections of 2K by 2K pixels show that TomoPy's iterative methods can scale up to thousands of cores on an IBM BG/Q supercomputer with almost perfect speedup and can reduce total reconstruction times for such datasets by more than 95.4% on 32K cores relative to 1K cores. Moreover, the average reconstruction times are improved from 2 hours (256 cores) to 1 minute (32K cores), thus enabling near-real-time use (Biçer et al., 2015).

2.2. The ASTRA toolbox

The ASTRA toolbox is an open-source software toolbox developed at the University of Antwerp, Belgium, and at the Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands, that is focused on the reconstruction of two-dimensional (slice-based) and three-dimensional tomographic datasets (van Aarle et al., 2015). The toolbox is available for Linux and Windows operating systems, and is aimed at providing a fast and flexible development platform for tomographic reconstruction algorithms. Because of its flexibility, it can be applied to various scanning geometries and acquisition modes, such as (bio)medical and industrial µCT (Plantagie et al., 2015), electron tomography (Roelandts et al., 2012), neutron tomography (Peetermans & Lehmann, 2013; Van Eyndhoven et al., 2015) and synchrotron tomography (Reischig et al., 2013). The toolbox uses CUDA for NVIDIA GPUs to perform accelerated parallel computations, reducing the computation time of many tomographic operations (Palenstijn et al., 2011). Most two-dimensional slice-based operations can also be run on standard CPUs, in which case the toolbox supports different projection kernels, i.e. ways of discretizing the projection operations. A comparison of various projection kernels can be found in the paper by Xu & Mueller (2006). When using GPUs, only a single set of projection kernels is supported for optimal performance. Through either a MATLAB or Python interface, the tomographic operations can be easily used and combined with other numerical code or software for pre-processing, post-processing or analysis of the acquired data. The toolbox also provides a matrix-like interface to linear tomography operators, allowing them to be easily used in existing and new code (Bleichrodt et al., 2015). A version of the ASTRA toolbox with MPI support for high-performance computing is available as a separate package (Palenstijn et al., 2015).

The ASTRA toolbox includes many popular tomographic reconstruction methods (see Table 1), such as the analytic filtered backprojection (FBP) method and the iterative SIRT method (Kak & Slaney, 2001) and CGLS method (Hansen, 1998). These methods support various parameters that can help improve reconstruction quality; for example, the choice of filter to use in the FBP method, and additional nonnegativity constraints in the SIRT method. An important feature of the ASTRA toolbox is that it also provides building blocks that can be used to develop advanced tomographic reconstruction methods. For example, using the optimized methods for the forward projection of objects and the backprojection of sinograms, it is possible to develop efficient advanced iterative methods, such as total variation regularized methods, using the ASTRA toolbox. A recent addition to the toolbox is a plugin system, which enables algorithm developers to easily distribute new tomographic reconstruction methods, and ASTRA users to easily install and use them with minimal changes in production scripts. With the TomoPy integration presented in this paper, these ASTRA plugins will automatically be usable in TomoPy as well.

2.3. Implementation

The code to integrate TomoPy and the ASTRA toolbox is written in the Python language, since TomoPy is mainly written in Python and the ASTRA toolbox includes a Python interface as well. Specifically, a first step was to add code to TomoPy that enables the use of other Python libraries to perform tomographic reconstruction instead of TomoPy's included algorithms. Using this new feature, code was added which enables the use of the ASTRA toolbox to perform the reconstruction. By integrating the ASTRA toolbox into TomoPy in this way, synchrotron users that are familiar with TomoPy will be able to use the ASTRA toolbox with minimal effort. Note that other tomographic reconstruction libraries that include a Python interface can be integrated in TomoPy in the same way. Since the integration requires both toolboxes to be installed on the same machine, it is only supported on operating systems in which both TomoPy and the ASTRA toolbox can be installed. Currently, this is only the case for modern Linux operating systems, but other operating systems will be supported once both TomoPy and the ASTRA toolbox support them.

In the interfacing code, the parallel-beam geometry defined by TomoPy (i.e. the number of detector pixels, the angles for which projections are acquired, and the center of rotation) is translated to a corresponding ASTRA geometry, and the chosen ASTRA reconstruction method is performed. Afterwards, the result of the reconstruction is stored in TomoPy memory, and all ASTRA objects are cleaned up. In this way, the reconstruction step is completely self-contained and independent of any pre-processing or post-processing step. An advantage of this independence is that user scripts do not have to be rewritten to use the ASTRA toolbox: only the reconstruction function call has to be modified. Also, changing between different reconstruction methods, between CPU and GPU implementations, and between different reconstruction parameters usually only requires changes in a single line of the user script. Examples of these minimal changes are shown in §3.2. A schematic overview of the full processing workflow from loading the raw data to analysis is shown in Fig. 1.

Figure 1 Schematic overview of the workflow of processing a dataset with the integrated TomoPy and ASTRA framework. Note that the reconstruction step can be performed by either TomoPy or the ASTRA toolbox.

3.1. Installation

Both TomoPy and the ASTRA toolbox can be installed using the Conda package management system (https://conda.pydata.org). The advantage of using Conda over other Python package management systems is that Conda allows for the inclusion of non-Python library dependencies, which are commonly needed for numerical toolboxes such as TomoPy and the ASTRA toolbox. Since the goal of both toolboxes is to be easily installable at the various workstations and computational clusters available at synchrotrons, which may each be running a different environment of installed libraries and library versions, the ability to tightly control the library dependencies is important to create a user-friendly installation process. To install TomoPy in a Conda environment, the following command can be used:

A similar command can be used to install the ASTRA toolbox:

Note that both toolboxes can be installed and used independently from each other. TomoPy will automatically detect whether the ASTRA toolbox is available, and enables the use of ASTRA methods if this is the case. Both toolboxes can also be compiled and installed from source code, which can be downloaded from their respective git repositories (https://github.com/tomopy/tomopy and https://github.com/astra-toolbox/astra-toolbox). Compared with installing using Conda, it is easier to make modifications and contribute to the development of the toolboxes when installing from source code, but the compilation step requires the availability of several library dependencies on the workstation.

3.2. Usage

We will now show how the new features can be used after installation of both toolboxes. The following example script shows a simple standard TomoPy workflow, loading data from disk, normalizing the data using the flatfield and darkfield images, and finally reconstructing with the standard TomoPy gridrec method (all scripts shown in this paper are available in the supporting information):

To modify this script to reconstruct using the ASTRA toolbox instead, only line 13 has to be changed, replacing gridrec with tomopy.astra, specifying which method to reconstruct with in the method option, and specifying which type of projection kernel to use in the proj_type option. An overview of common options that are used when reconstructing with the ASTRA toolbox is given in Table 2. For example, to reconstruct with FBP using a voxel-driven kernel, we change the final part of the above script to:

Table 2 List of common options that are used when reconstructing with the ASTRA toolbox through TomoPy Option Description Example values Which reconstruction method to use Which projection kernel to use Number of iterations to use in iterative method Python dictionary with extra method-specific options List of GPU indices to use for reconstruction

When running on a machine with a GPU with CUDA capabilities, the same reconstruction can be performed using optimized GPU code, greatly decreasing the needed computation time. This can be realised by specifying cuda as the projection kernel, and use a GPU-enabled method (FBP_CUDA):

Iterative methods can be used by specifying the corresponding ASTRA method (e.g. CGLS_CUDA for a GPU-enabled CGLS method), and the number of iterations to use in the num_iter option:

Most reconstruction methods in the ASTRA toolbox support several parameters that can help improve reconstruction quality. In the TomoPy integration, these parameters are specified by supplying them in the extra_options setting. For example, to add a nonnegativity constraint to the GPU-enabled SIRT method, we add MinConstraint to the extra_options setting (note that lower bounds other than zero can also be used):

An overview of the various parameters that are supported by the reconstruction methods can be found on the website of the ASTRA toolbox (https://www.astra-toolbox.com).

If multiple GPUs are installed in the workstation running TomoPy and the ASTRA toolbox, the computations can be distributed over multiple GPUs by specifying a list of GPU indices in the gpu_list option. Since each slice of the reconstruction can be computed independently from the other slices, a significant reduction of computation time can be achieved by distributing the computations in this way. For example, to use four installed GPUs, labeled 0 through 3, we use [0, 1, 2, 3] as the gpu_list:

Finally, ASTRA plugins can be used by first registering them with the ASTRA toolbox itself, and using the method name defined by the plugin as the method option. Extra parameters for the reconstruction can be specified using the extra_options setting, similar to standard ASTRA methods. An ASTRA plugin is typically distributed as a Python class within a Python package, which has to be imported separately. After importing, the astra.plugin.register method is used to register a plugin with the ASTRA toolbox. For example, suppose that there is a plugin class tvtomo.plugin within the tvtomo package, with the method name TV-FISTA and an additional parameter tv_reg. To use this plugin in TomoPy, the following code can be used:

3.3. Computation time

In Fig. 2, a comparison is shown between the computation times per slice of reconstructions computed with different methods of both TomoPy and the ASTRA toolbox, for a single slice of a dataset with 1200 detector pixels and 1024 projections. All reconstructions were computed on a workstation with 128 GB of memory, two Intel Xeon E5-2623 v3 CPUs (four cores each, running at 3.0 GHz) and two Geforce GTX Titan Z cards (running at their default clock speeds). Each Titan Z card contains two GPUs, with 15 multiprocessors (2880 cores) and 6 GB of memory per GPU. The workstation was using the Fedora 21 operating system, with the official CUDA 7.0 drivers provided by NVIDIA. We compare the computation times of gridrec computed using TomoPy, FBP computed using the ASTRA toolbox, and 100 iterations of SIRT computed using both TomoPy and the ASTRA toolbox. For the CPU-based methods of TomoPy and the ASTRA toolbox, computation times are shown for using both a single core and all eight cores of the machine. For the GPU-based methods of the ASTRA toolbox, computation times are shown for using both a single GPU and all four installed GPUs.

Figure 2 Computation time per slice of reconstructing with the gridrec method computed with TomoPy, the FBP method computed with the ASTRA toolbox, and 100 iterations of the SIRT method computed with both, for 1024 projections and a detector width of 1200 pixels. Results are shown for using a single CPU core, 8 CPU cores, a single GPU, and 4 GPUs.

The results of Fig. 2 show that for the SIRT method a significant reduction of computation time can be achieved by using GPUs instead of CPUs for computation, with the computation time per slice when using TomoPy and eight CPU cores being roughly 600 times the computation time when using the ASTRA toolbox and four GPUs. Note that this reduction of computation time can be important in practice, since, in this case, computing the SIRT reconstruction of 200 slices would take less than 4 minutes with four GPUs, compared with more than 32 hours with eight CPU cores. Therefore, by using GPU-based methods, it is possible to compute iterative reconstructions during experiments, enabling direct inspection of the reconstructions and the possibility of making adjustments to improve the experimental results during the experiment itself.

In contrast to the iterative SIRT method, TomoPy's CPU-based gridrec method takes less time to compute compared with the GPU-based FBP method, especially when using multiple CPU cores. This is expected for problems with a relatively large number of projections, since the most costly computations of the gridrec method are the two-dimensional Fourier transforms, for which the computation time is independent of the number of projections. On the other hand, in the FBP method the most costly computation is the backprojection operation, for which the computation time scales linearly with the number of projections. Note that the reconstruction quality of gridrec and FBP reconstructions are usually similar (Marone & Stampanoni, 2012).