Elsevier

Medical Image Analysis

Volume 13, Issue 2, April 2009, Pages 325-342
Medical Image Analysis

Deterministic and probabilistic approaches for tracking virus particles in time-lapse fluorescence microscopy image sequences

https://doi.org/10.1016/j.media.2008.12.004Get rights and content

Abstract

Modern developments in time-lapse fluorescence microscopy enable the observation of a variety of processes exhibited by viruses. The dynamic nature of these processes requires the tracking of viruses over time to explore spatial–temporal relationships. In this work, we developed deterministic and probabilistic approaches for multiple virus tracking in multi-channel fluorescence microscopy images. The deterministic approaches follow a traditional two-step paradigm comprising particle localization based on either the spot-enhancing filter or 2D Gaussian fitting, as well as motion correspondence based on a global nearest neighbor scheme. Our probabilistic approaches are based on particle filters. We describe approaches based on a mixture of particle filters and based on independent particle filters. For the latter, we have developed a penalization strategy that prevents the problem of filter coalescence (merging) in cases where objects lie in close proximity. A quantitative comparison based on synthetic image sequences is carried out to evaluate the performance of our approaches. In total, eight different tracking approaches have been evaluated. We have also applied these approaches to real microscopy images of HIV-1 particles and have compared the tracking results with ground truth obtained from manual tracking. It turns out that the probabilistic approaches based on independent particle filters are superior to the deterministic schemes as well as to the approaches based on a mixture of particle filters.

Introduction

Spatial–temporal exploration of dynamic processes in virus–cell interactions is important for advancing our understanding of viral infections. For instance, information on the dynamics exhibited by the enveloped human immunodeficiency virus (HIV-1) upon entry into a host cell can yield insight on the action mechanisms of antiviral drugs. While traditional bulk measurements yield only limited information on the dynamics of the viral entry process, the determination of trajectories of single viruses provides the unique opportunity to analyze the dynamics of individual events (Brandenburg and Zhuang, 2007). The general aim of our work is to study the dynamic behavior of HIV viruses based on live-cell microscopy using double fluorescently labelled virus particles designed for a detailed analysis of the cell entry process (Müller et al., 2004, Lampe et al., 2007). This labeling technique results in two-channel images, where the two different labels within one virus particle allow to distinguish complete particles before cell entry from those that have undergone fusion with the cell membrane. However, to draw meaningful and statistically sound conclusions about the virus entry process it is necessary to evaluate a large number of individual events. Thus, a large amount of visual data needs to be analyzed. Therefore, automatic tracking techniques are required to quantify the dynamic behavior of viruses.

However, the task of virus tracking is challenging. In particular, the small size of viruses is a problem, since, due to the limited spatial resolution of the microscope, a virus is displayed as a small ‘particle’ that yields little visual information (e.g., salient visual features) to support the localization and identification of the virus. The small size of viruses also entails that a limited number of fluorescent molecules can be attached to them. The resulting low contrast of each particle hinders its identification against the cellular autofluorescent background. Furthermore, their complex motion behavior, which includes abrupt changes in speed and direction, precludes the usage of motion constraints (e.g., constraints concerning the direction or smoothness of the motion) employed in previous tracking applications. Virus particles may also move out of the focal plane during observation, which leads to a change in appearance (size and fluorescence intensity). Additionally, the large number of viruses in the image sequences rules out algorithms that are only applicable to one object or few objects. Finally, a relatively low signal-to-noise ratio (SNR) hinders the accurate localization of particles.

Tracking comprises generally two main tasks: (1) object representation and localization, as well as (2) spatial–temporal filtering and motion correspondence (Comaniciu et al., 2003). In previous work on virus tracking, a deterministic two-step paradigm encompassing the steps of virus localization and motion correspondence has been employed. For virus localization, most algorithms employ a maximum intensity search strategy, in which the positions of the viruses are associated with intensity peaks. Techniques based on intensity thresholds (e.g., Lakadamyali et al., 2003) or intensity moments of detected candidate viruses (e.g., Sbalzarini and Koumoutsakos, 2005) have been employed for rejecting noise-induced maxima. The position of the particles may be refined by model fitting (e.g., Anderson et al., 1992, Seisenberger et al., 2001). For motion correspondence, a nearest neighbor model is typically employed. However, in image regions with a high density of viruses, the search for correspondences becomes ambiguous, since several possibilities are plausible. To address this issue, approaches that consider the motion of all viruses between two consecutive time steps via graph-theoretical algorithms have been introduced (e.g., Schutz et al., 1997, Sbalzarini and Koumoutsakos, 2005). While being computationally efficient, the two-step tracking paradigm obviates a spatial–temporal filtering step and therefore incurs in relative loss of performance under difficult imaging conditions (e.g., non-homogeneous background, large number of objects, low SNR).

Probabilistic tracking approaches are generally characterized by the inclusion of a filtering step (e.g., Anderson and Moore, 1979). By modeling the spatial–temporal behavior of an object using a dynamical model and incorporating measurements derived from the images via a measurement model, the filter estimates the object’s position given a series of such measurements. For this task, the Kalman filter (Kalman, 1960) can be employed if the dynamical and measurement models are represented by linear systems excited with Gaussian noise processes. In the case of virus tracking, Arhel et al., 2006, Genovesio et al., 2006 have devised an approach based on a pool of Kalman filters integrated via an interacting multiple model (IMM) filter in combination with a multi-scale detection algorithm for virus localization and a probabilistic split-and-merge mechanism for motion correspondence. However, the object localization and filtering steps are uncoupled, which entails that the object localization algorithm disregards temporal information, and likewise the filtering technique does not analyze directly the image data, which diminishes the effectiveness of the overall approach. An integrated approach exploiting the spatial–temporal nature of tracking is supposed to yield better results.

The particle filter (e.g., Gordon et al., 1993, Isard and Blake, 1998) is a sequential Monte Carlo method that exploits more efficiently the dynamical information encoded in an image sequence. In contrast to the Kalman filter, the particle filter can employ non-linear, non-Gaussian models that provide a high degree of robustness in cases of unpredictable motion and complicated imaging situations, e.g., cluttered images (Doucet et al., 2001). However, using particle filters for multiple object tracking remains a challenging issue. Existing approaches can be classified into particle filters defined on a multiple-body state space, where the entire configuration of all objects is jointly estimated, and particle filters defined on a one-body state space, where the configuration of each object is independently estimated. The former approach, known as joint particle filters (e.g., MacCormick and Blake, 2000, Isard and MacCormick, 2001, Tweed and Calway, 2002), is quite suitable for modeling interactions and occlusions, yet, without alternative sampling strategies (e.g., Khan et al., 2005), it is only applicable to few objects due to the computational effort that is involved in estimating the posterior distribution in a high-dimensional space. The latter, denoted as independent particle filters (e.g., Qu et al., 2005, Cai et al., 2006), while inducing lower computational requirements, tend to fail as objects pass close to each other, since the independent filters latch to the object with the best likelihood (Khan et al., 2005).

Particle filters have been recently introduced into biological imaging (for other approaches for tracking biomolecular structures see, e.g., Tvarusko et al., 1999, Thomann et al., 2002, Yang et al., 2005, Racine et al., 2006). In Cui et al. (2006), a particle filter has been used to track a single leukocyte (white blood cell). In Smal et al. (2006), a mixture of particle filters has been employed to track microtubuli (cellular line structures). Neither of the approaches is fully automatic, since the user has to manually determine the position of the objects in the first image. Work on a fully automatic tracking approach based on particle filters has been presented in Smal et al., 2007, Godinez et al., 2007. In Smal et al. (2007), a mixture of particle filters is used to track microtubuli. Object localization is achieved via a spatial probability density function, which is generated by normalizing the intensity values of Gaussian filtered images. A nearest neighbor approach is utilized for motion correspondence. To address the problem introduced by the proximity of objects, the approach employs a Markov random field that prevents objects from getting too close. Preventing proximity, however, might be undesirable if objects really merge. The approach has been applied to image sequences consisting of 20 frames. In Godinez et al. (2007), a mixture of particle filters is used to track virus particles. In comparison to Smal et al. (2007), this probabilistic approach employs a different motion model and a more accurate localization algorithm, namely 2D Gaussian fitting (Cheezum et al., 2001), which yields more accurate parameter estimates for the employed appearance model. The applicability of the approach has been demonstrated for tracking multiple viruses in relatively long image sequences (e.g., more than 200 frames).

In this paper, based on our previous work in Godinez et al. (2007), we introduce probabilistic and deterministic approaches for tracking multiple viruses in microscopy time-lapse images. The developed approaches are fully automatic and can handle multi-channel microscopy images. We have developed four probabilistic approaches based on particle filters, namely two approaches using a mixture of particle filters (MPF) and two approaches using independent particle filters (IPF). The main advantage of both particle filter techniques is their relatively low computational requirements, which is a key property in our application, given the large number of virus particles in a relatively large number of images. In contrast to the approaches based on MPF, the performance of the approaches based on IPF does not deteriorate as the number of objects increases. In addition, the problem of filter coalescence (i.e., merging of filters) that arises when using IPF is addressed via a penalization mechanism based on a deterministic motion correspondence algorithm. In comparison to previous approaches, our penalization scheme allows temporary merging of objects, which is a desirable property, since the trajectories of virus particles may merge for a certain time period. Moreover, in comparison to previous particle filter approaches for particle tracking in biological images, we employ a more accurate localization algorithm based on 2D Gaussian fitting. We have also developed two different deterministic two-step approaches, namely two approaches based on either an enhanced 2D Gaussian fitting algorithm or the spot-enhancing filter for particle localization, in combination with a global nearest neighbor scheme for motion correspondence. The developed approaches have been applied to synthetic image sequences containing virus-like objects as well as to real microscopy image sequences containing HIV-1 particles and autofluorescent structures of the target cells. We have also performed an experimental comparison with two Kalman filter approaches. Thus, in total we have evaluated eight different tracking approaches. To the best of our knowledge, it is the first time that an approach based on particle filters is employed for virus tracking in fluorescence microscopy image sequences. Concerning the approaches based on IPF, our motion correspondence algorithm guarantees an optimal assignment (in terms of the employed distance) between the tracked objects and the position measurements generated by the localization algorithms. Moreover, we apply our approaches to relatively long image sequences.

This paper is organized as follows. First, we describe the deterministic approaches (Section 2). Subsequently, we provide a description of our probabilistic approaches in Sections 3 Probabilistic tracking approaches, 4 Model of virus particles. Section 5 presents experimental results for synthetic and real image sequences. Concluding remarks are given in Section 6.

Section snippets

Object localization by spot-enhancing filter

In fluorescence microscopy images, the intensity structure corresponding to a virus particle resembles a 2D Gaussian function, where the peak intensity of the particle is larger than the intensity of the background. One approach for localizing such particles is threshold-based segmentation. However, due to image noise and other biological structures with similar intensities (e.g., cellular autofluorescence), this approach generally leads to a high number of false detections. To improve object

Spatial–temporal filtering

The theory of sequential state estimation (e.g., Anderson and Moore, 1979) assumes that an object is represented by a (hidden) state vector xt and that a noisy measurement yt reflects the true state xt. At time step t, the aim is to estimate the state xt given a sequence of measurements y1:t. A Bayesian approach to the problem is to estimate the posterior probability density function (pdf) p(xt|y1:t), which represents a degree of “belief” on the state xt given a series of measurements y1:t. To

Model of virus particles

In this section, we describe the appearance, dynamical, and measurement models that we have employed in our particle filter approaches (for both the MPF and IPF).

Experimental results

The approaches described above have been applied to synthetic as well as real microscopy image sequences. The following six tracking schemes have been used: (1) spot-enhancing filter in combination with a global nearest neighbor scheme (SEF&GNN), (2) spot-enhancing filter and a mixture of particle filters (SEF&MPF), (3) spot-enhancing filter and independent particle filters (SEF&IPF), (4) 2D Gaussian fitting and a global nearest neighbor scheme (GaussFit&GNN), (5) 2D Gaussian fitting and a

Conclusions

We have introduced fully automatic deterministic and probabilistic approaches for virus tracking in time-lapse fluorescence microscopy images. The deterministic approaches follow a classic two-step paradigm, while the probabilistic approaches are based on particle filters. Our experiments for both synthetic and real image sequences indicate that the performance of the deterministic approaches is not very accurate under realistic imaging situations (e.g., spurious particles). This arises mainly

Acknowledgement

This work has been supported by the BMBF (FORSYS) project VIROQUANT.

References (45)

  • B. Brandenburg et al.

    Virus trafficking-learning from single-virus tracking

    Nat. Rev. Microbiol.

    (2007)
  • Cai, Y., de Freitas, N., Little, J.J., 2006. Robust visual tracking for multiple targets. In: Proceedings of European...
  • L.S. Churchman et al.

    Single molecule high-resolution colocalization of Cy3 and Cy5 attached to macromolecules measures intramolecular distances through time

    Proc. Natl. Acad. Sci. USA

    (2005)
  • D. Comaniciu et al.

    Kernel-based object tracking

    IEEE Trans. Pattern. Anal. Mach. Intell.

    (2003)
  • I.J. Cox

    A review of statistical data association for motion correspondence

    Int. J. Comput. Vision

    (1993)
  • A. Doucet et al.

    An introduction to sequential Monte Carlo methods

  • A. Genovesio et al.

    Multiple particle tracking in 3-d+t microscopy: method and application to the tracking of endocytosed quantum dots

    IEEE Trans. Image Process.

    (2006)
  • Godinez, W.J., Lampe, M., Wörz, S., Müller, B., Eils, R., Rohr, K., 2007. Tracking of virus particles in time-lapse...
  • N. Gordon et al.

    Novel approach to non-linear/non-Gaussian Bayesian state estimation

    IEE Proceedings F, Radar and Signal Processing

    (1993)
  • R. Haralick et al.
    (1992)
  • M. Isard et al.

    CONDENSATION – conditional density propagation for visual tracking

    Int. J. Comput. Vision

    (1998)
  • M. Isard et al.

    BraMBLe: a Bayesian multiple-blob tracker

  • Cited by (93)

    • An improved high accuracy PTV algorithm for pebble flow

      2021, Powder Technology
      Citation Excerpt :

      The global linking methods such as Spatiotemporal tracing [51] and graph-based optimization approaches [52,53] were applied to improve matching accuracy. Various Bayesian estimation approaches have been explored in recent years [54,55]. Meijering et al. [56] summarized 30 currently open source available tools for cell and/or particle tracking.

    • PySTACHIO: Python Single-molecule TrAcking stoiCHiometry Intensity and simulatiOn, a flexible, extensible, beginner-friendly and optimized program for analysis of single-molecule microscopy data

      2021, Computational and Structural Biotechnology Journal
      Citation Excerpt :

      Away from STORM/PALM-type static reconstruction, many codes have been developed to find individual foci in noisy live-cell microscopy data. In general, classical algorithms in the same class as PySTACHIO and ADEMSCode operate through identification of local intensity maxima, though some include pre-filtering steps such as Gaussian filtering [23–27], Laplacian of Gaussian [25,26,28], wavelet products [29,30], or deconvolution [31]. In general, a functional form is then fit to detected peaks (commonly Gaussian but occasionally Lorentzian [32]), though in some cases localization itself is done using adaptive thresholding methods [27].

    View all citing articles on Scopus
    View full text