Embracing the uncertainty: the evolution of SOFI into a diverse family of fluctuation-based super-resolution microscopy methods

Fluorescence microscopy relies on detecting light emitted from the sample after photo-excitation of fluorophores to a higher energy state with a light source such as a lamp or a laser. Absorption of light is followed by vibrational or conformational relaxation and emission of light at a longer wavelength. In a standard epifluorescence microscope (figure 1(a)), the same objective transmits the illuminating beam to the object and collects the emitted fluorescence. A dichroic mirror and dielectric filter separate the faint fluorescence from scattering and reflection of the illumination light and an additional lens (often referred to as a tube lens) images it onto a camera.

Zoom In Zoom Out Reset image size

The resolution of an image captured with an optical microscope is limited by diffraction. This stems from the wave-like character of light: no matter what kind of optical components we use, we cannot focus the light waves beyond about half of its wavelength. As, a result, the image of a point object is not a single point but a finite-sized blur that we term the point spread function (PSF). The shape of the PSF depends on the properties of the imaging system and can be calculated knowing its geometry. In particular, the PSF of a circular lens is an Airy disc—a circular pattern with a bright center spot and alternating bright and dark rings [36]. The first dark ring appears at a radius of $\sim \frac{0.61\lambda}{\mathrm{NA}}$, where λ is the wavelength of light and $\mathrm{NA}$ is the numerical aperture of the lens, described in the following paragraph. The Rayleigh criterion attempts to provide a simple quantitative metric to the complex concept of resolution. According to it, two imaged objects can be distinguished from each other if the distance between them is longer than the radius of the first dark ring in the Airy disc. Otherwise, the Airy discs merge together and cannot be resolved.

We now present a complementary discussion, generalizing the topic of resolution and its enhancement in terms of spatial frequency content, i.e. the Fourier transform of the object or image. Because of its finite size, an objective lens collects light rays emerging from the sample up to some maximal angle, θ₀, from which we derive the numerical aperture, $\mathrm{NA} = n\cdot \sin\theta_0$ (n is the refractive index of the medium). In a simplified yet constructive picture, this angle directly translates to the maximal spatial frequency that propagates from the object to the image, $k^{^{\prime}}_0 = \frac{2\pi}{\lambda}\cdot \mathrm{NA}$. Higher spatial frequency content is lost in this propagation and as a result the finer details of the object do not translate to the image. In image processing, one would refer to this process as a short-pass filter with a cut-off frequency of $k^{^{\prime}}_0$. Putting it all together, to simulate the imaging system, the object should be Fourier transformed, its content set to zero at frequencies beyond $k^{^{\prime}}_0$ and finally inverse Fourier-transformed back into real space. A thorough, yet intuitive, explanation for this mechanism is given in the review in reference [37].

A fluorescence image is proportional to the light intensity (incoherent), a time-average of the electric field squared. In the Fourier domain, this squaring translates to a convolution of the frequency filtered PSF with itself, containing spatial frequencies up to $k_0 = 2k^{^{\prime}}_0$. More accurately, to calculate the fluorescence intensity image, we multiply the frequency content of the sample with the modulation transfer function (MTF) which becomes zero for $k\gt k_0$, the cut-off frequency. Unlike the simplified coherent case, the MTF strongly depends on k even within the $k\lt k_0$ region, suppressing the contribution of higher spatial frequencies (see figure 3(a)). This dependence of the MTF on k will be discussed in greater detail in section 2.4. Circling back to the image position space, the PSF is simply the Fourier transform of the MTF. Thus, extending the frequency cut-off translates to a narrower PSF and higher resolution.

Super-resolution microscopy can be framed as an attempt to narrow the image PSF (or extend the MTF cut off) beyond the Abbe diffraction limit and therefore resolve features that are closer together than $\sim \frac{\lambda}{2}$. The 2014 Nobel prize in Chemistry was awarded for the development of two groundbreaking super-resolution methods that revolutionized far-field light microscopy. The earliest, STED, uses the non-linear response of labeling emitters to light by saturating their excitation depletion with a doughnut-shaped beam. As a result, only emitters at the very confined space at the center of the scanned doughnut contribute to the signal and the effective PSF is significantly shrunk.

Perhaps the most wide-spread approach for super-resolution, SMLM, relies on fluorescence fluctuation of emitters. These randomly shine for a short enough period of time so that they mostly do not overlap with each other in any frame. Collecting the precise position of these emitters over a long video produces a substantially sharper image than the averaged image. The following sub-section describes the PSF narrowing mechanism of another successful super-resolution method that relies on fluorescence fluctuations, SOFI—the subject of this review.

2.2.1. Second-order SOFI

In practice, the intensity of light emitted from a fluorophore is typically not constant in time. Repeated transitions between a fluorescent and non-fluorescent state lead to intermittent emission or blinking that can be exploited for obtaining sub-diffraction images. SOFI relies on recording a movie of a sample and analysing the changes of its fluorescence intensity in time. This image series is then processed into a single correlation image with increased resolution.

Consider a two-dimensional sample consisting of N independent emitters whose time-dependent brightness is given by $O(\mathbf{r},t) = \sum_{k = 1}^N\epsilon_k\cdot{s_k(t)}{\cdot}\delta(\mathbf{r}-\mathbf{r}_k)$, where r_k is the emitter's location, $\epsilon_k$ is the constant molecular brightness and $s_k(t)$ is the time-dependent component with values between 0 and 1 (figure 1(b)).

The time-dependent fluorescence image at position r is a convolution of the microscope's PSF $U(\mathbf{r})$ with the sample brightness, $O(\mathbf{r},t)$ (figure 1(c)):

$$\begin{equation} F(\mathbf{r},t) = O(\mathbf{r},t)*U(\mathbf{r}) = \sum_{k = 1}^NU(\mathbf{r}-\mathbf{r}_k)\cdot\epsilon_k{\cdot}s_k(t). \end{equation} \tag{ 1 }$$

We explicitly assume here that the positions of emitters do not change during the measurement and temporal changes are caused only by blinking.

Wide-field microscopy employs time average over $F(r,t)$, which is a sum of averaged contributions from each emitter multiplied by $U(\mathbf{r}-\mathbf{r}_k)$. Contrariwise, a SOFI image is derived from correlations of the time dependent signal (figure 1(d)). In the most basic form of SOFI, it can be shown [22] that the second order auto-correlation function $G_2(\mathbf{r},\tau)$ already provides a $\sqrt{2}$ improvement in resolution (figure 1(e)). To mathematically observe that we can write the full expression for the auto-correlation of the fluorescence intensity:

$$\begin{equation} G_2(\mathbf{r},\tau) = \langle{\delta{F}(\mathbf{r},t+\tau)\cdot{\delta{F}(\mathbf{r},t)}}\rangle = \sum_{k = 1}^{N}U^2(\mathbf{r}-\mathbf{r}_k)\cdot\epsilon^2_k\cdot\langle{\delta s_k(t){\cdot}\delta{s}_k(t+\tau)}\rangle, \end{equation} \tag{ 2 }$$

where $\langle..\rangle$ denotes time averaging and $\delta{F}(t) = F(t)-\langle{F}\rangle$ describes the fluctuation, i.e. the difference with respect to the average intensity at a given time. Note that, we assume here that emitters fluctuate independently of one another, i.e. the emission of any two emitters is uncorrelated in time. As a result, cross terms proportional to the product of two different emitters' intensities $\langle{\delta s_i(t){\cdot}\delta{s}_j(t+\tau)}\rangle$, for i ≠ j, average to zero and equation (2) contains only a single sum over emitters.

The key difference between the correlation contrast in equation (2) and the wide-field contrast in equation (1) is the presence of a squared PSF. Raising the PSF to a higher power results in the appearance of sharper features in the image, hence super-resolution (see figure 2(a)). In particular, for a Gaussian-shaped PSF (often a good approximation), squaring the PSF corresponds to a reduction of width by a factor of $\sqrt{2}$. Moreover, this reduction happens in all three dimensions, thus increasing the resolution of the SOFI image in 3D. A comprehensive discussion on 3D resolution is included in section 3.1. An additional advantage of SOFI, reflected in equation (2), is the fact that any non-fluctuating background signal vanishes on average.

Zoom In Zoom Out Reset image size

The reader may recall (section 2.1) that squaring the PSF can be interpreted as a convolution of the MTF, the Fourier transform of the PSF, with itself. Indeed, the auto-correlation image (equation (2)) includes a squaring of a PSF in real space and therefore corresponds to convolution in the Fourier domain and an extension of the spatial frequencies cut-off up to $2k_0$ (see the MTF simulation in figures 3(a) and (b)). The presence of higher spatial frequencies results in narrower features in the image and is therefore a complimentary indication of super-resolution (see figures 3(f) and (g)). While these features hint that a super-resolved image can, in principle, be reconstructed, some caution is required since the mere presence of higher frequencies does not always mean that the image is true to the original sample structure and artifacts may create distortions. The spatial frequency point-of-view on resolution will become critically important when discussing higher correlation orders in the following section and deconvolution in section 2.4.

Zoom In Zoom Out Reset image size

2.2.2. Higher-order SOFI

The above description can be generalized to higher correlation orders. These contain frequency components beyond the cut off of the second-order correlations and therefore have the potential to increase the resolution even further. However, the presence of the cross-terms (multiplications of PSFs from different emitters) may yield sharp features that do not reflect the shape of the underlying object. To remedy this, SOFI analysis uses cumulants, rather than correlations, eliminating the contribution of cross-terms. The nth order cumulant, a linear combination of correlations up to the nth order, is described by a recursive formula given in reference [38]. Once cross terms have been eliminated, the resulting nth order SOFI image is given by:

$$\begin{equation} C_n(\mathbf{r},\tau_1,...,\tau_{n-1}) = {\sum_k}U^n(\mathbf{r}-\mathbf{r}_k)\epsilon^n_k{w_{n,k}(\tau_1,...,\tau_{n-1})}, \end{equation} \tag{ 3 }$$

where $w_{n,k}(\tau_1,{\ldots},\tau_{n-1})$ is the single-emitter cumulant of the nth order.

Analogous to the second-order expression, equation (3) contains the PSF raised to the power of n. This implies that for a Gaussian PSF, the resolution improvement is by a factor of $\sqrt{n}$ (see figures 2(a) and (b)). Although, in theory, there is no limit on the calculated cumulant order and the corresponding resolution improvement, there are in fact, some practical aspects that limit the application of higher-order SOFI. In general, higher orders require longer measurements and are more sensitive to noise. This, in turn, entails increasing the exposure time, illumination intensity or number of acquired frames. All these changes have significant drawbacks including higher fluorescence bleaching, susceptibility to sample drift. Moreover, small variations in molecular brightness, $\epsilon_k$, also raised to the nth power, are translated to an increasing signal range with higher orders resulting in the image being dominated by the brightest emitters. Finally, there are some considerations specific to SOFI such as artifacts resulting from different blinking statistics of neighbouring fluorophores. These topics are discussed in detail in section 5. Recent theoretical analyses show that it is not trivial to quantify the amount of added information in fluorescence cumulants of different orders [39, 40].

Until now we have discussed auto-cumulant-based resolution enhancement, obtained through computing temporal correlations at a single position. This description can be generalized to cross-cumulants obtained from spatio-temporal correlations. Let us first consider the second-order cross-cumulant between positions r₁ and r₂ for a Gaussian-shaped PSF [41]:

$$\begin{equation} \begin{split} C_2(\mathbf{r_1},\mathbf{r}_2,\tau) & = {\sum_{k = 1}^N}U(\mathbf{r}_1-\mathbf{r}_k)U(\mathbf{r}_2-\mathbf{r}_k)\epsilon^2_k{w_k(\tau)} \\ & = U\left( \frac{\mathbf{r}_1-\mathbf{r}_2}{\sqrt{2}} \right) {\sum_{k = 1}^N}U^2\left(\frac{\mathbf{r}_1+\mathbf{r}_2}{2}-\mathbf{r}_k\right)\epsilon^2_k{w_{2,k}(\tau)} \end{split}. \end{equation} \tag{ 4 }$$

Two things stand out here. First, a cross-cumulant between $\mathbf{r_1}$ and r₂ results in a signal in the geometric center of these two points $\left(\frac{\mathbf{r_1}+\mathbf{r_2}}{2}\right)$. Second, the expression is multiplied, or weighted by a 'distance factor' $U\left(\frac{\mathbf{r_1}-\mathbf{r_2}}{\sqrt{2}}\right)$. The latter can be understood intuitively: since emitters fluctuate independently, only points within the same emission PSF can contain contributions from the same emitter and have a non-zero correlation.

The significance of a signal generated at the geometric center of the two positions becomes apparent when we consider a real-life camera with discrete, finite-sized pixels. If we acquire an image series and calculate auto-cumulants of increasing orders, eventually the pixel size will limit further resolution improvement. In contrast, by calculating cross-cumulant of neighboring pixels, we create additional 'virtual pixels' and increase the pixel density. Contrary to a simple interpolation, these pixels carry additional information. The process of creating virtual pixels is further discussed in section 5.1.

Similarly as for auto-cumulants, equation (4) can be extended to higher orders, up-sampling the image even further. The nth order cross-cumulant will theoretically yield an n × n-fold up-sampled SOFI image [41].

Fourier reweighting (FR), sometimes also termed Wiener-deconvolution or Wiener-filtering, is a post-processing step that attempts to stretch the resolution of an image to its optimum. Its application to further enhance SOFI images was proposed very soon after SOFI's initial demonstration [41].

In order to describe FR and its benefits, it is helpful to look at the topic of image resolution from the spatial-frequency domain point-of-view, introduced in section 2.1. The PSF of an nth order SOFI image contains spatial frequencies up to $n{\cdot}k_0$, where k₀ is the spatial frequency cut-off of the diffraction-limited image, with higher frequencies corresponding to sharper features in the image. In contrast, the resolution of SOFI increases at a much less favorable scaling of $\sqrt{n}$. To understand this discrepancy, we look beyond the frequency cut-off and observe the functional form of the MTF, the Fourier transform of the fluorescence (incoherent) PSF (see section 2.1). To illustrate this delicate point, simulated MTFs (500 nm wavelength, 1.4 $\mathrm{NA}$) are presented in figures 3(a)–(e) (cross-sections in figure 3(k)) for diffraction limited imaging and different orders of SOFI and their FR version. The cut-off frequency, beyond which the MTF becomes identically zero, for each method is denoted as a white dashed circle whose radius grows linearly with the order. However, the gradient with which MTF approaches zero is quite different for the different methods: the decrease of amplitude with spatial frequency is more rapid for 2nd order SOFI than it is for the diffraction limited version and is even more rapid for the 4th order SOFI. For example, comparing the simulated MTF of a diffraction limited image (3(a)) and that of 2nd order SOFI MTF (3(b)), one can observe that while the cut-off frequency (white dashed circle) increases by a factor of 2, the reasonable amplitude portion (e.g. normalized amplitude greater than 0.1) did not expand as much and appears further away from the cut-off. This gap between the frequency cut-off and frequencies with reasonable amplitude MTF grows even further when observing 4th order SOFI (3(d)). As a result, the resolution scales in a sub-optimal $\sqrt{n}$ manner with the order in non-FR versions of SOFI [41] and ISM [35].

Theoretically, this can be amended by simply multiplying the Fourier transform of the image with the inverse of the MTF for the appropriate SOFI order, $W_n(k) = \frac{1}{\mathrm{MTF}_n(k)}$, for any $k\lt n{\cdot}k_0$. In practice, however, one has to consider the changes to signal-to-noise ratio (SNR) in the final image. Roughly speaking, SNR is the ratio between the measured signal level (e.g. pixel value in a camera image) and the average deviation under multiple repetitions of the measurement. Typically, SNR increases with the level of signal. In the Fourier domain, the MTF sharply decreases with frequency, corresponding to a lower signal and a lower SNR at high frequencies. Digitally amplifying those higher frequencies enhances the resolution, but at the same time increases the content of noise in the image. A standard approach to manage the resolution-SNR trade-off is using a Wiener-filter in the Fourier domain:

$$\begin{equation} W_n(k) = \begin{cases} \frac{1}{\mathrm{MTF}_n(k)+ \xi}& k\lt n{\cdot}k_0\\ 0& \textrm{otherwise}\\ \end{cases}, \end{equation} \tag{ 5 }$$

where k is the spatial frequency, and the parameter ξ is typically manually adjusted, using higher values for low SNR images. Here, a direct relation is created between the SNR of the image and its resolution; for a low SNR image a high value of ξ is required and the amplification of higher frequencies is moderate. Thus, nth order SOFI image may not achieve the potential × n resolution increase. A simulated demonstration of the resolution improvement through FR (inspired by [41]), for diffraction limited image, 2nd and 4th order SOFI and their FR versions, is given in figures 3(f)–(i).

Some practical consideration should be taken when applying FR. Estimating the PSF of an imaging system, experimentally or theoretically, is not a trivial task and errors in estimation would lead to image distortions. In theory, given an independent source of added noise, ξ would be the frequency dependent inverse of the image SNR [42]. Nonetheless, in low signal fluorescence microscopy camera noise is often not additive (e.g. shot noise) and the SNR is difficult to estimate. Using a constant parameter or a linearly increasing function of k have been shown to achieve near optimal resolutions expected for high SNR images [25, 35, 43].

In the previous sections we have described the principles and basic properties of SOFI. From a practical point of view, it is natural to ask how SOFI compares to other already established super-resolution imaging methods. A detailed comparison of different methods can be found in numerous reviews, including some recent ones that take into account latest developments in the field [37, 44, 45]. Here we will provide a short summary to give the reader an idea of where SOFI lies in the super-resolution microscopy landscape.

There are many criteria by which super-resolution imaging methods can be compared. The most obvious one is resolution, both lateral and axial. SOFI can offer lateral resolution improvement beyond the x2 that ISM and linear SIM can provide and an equivalent improvement to that of NSIM. However, although theoretically SOFI PSF narrowing can be arbitrarily large, if high-order cumulants are computed, the resolution achieved in practice is not as high as that of STED or SMLM, where molecules can be localized with lateral resolution close to their size, i.e. 10–20 nm. Axial resolution is a separate issue. Many of the early super-resolution microscopy approaches focused on thin samples and operated only in lateral direction, offering little or no improvement of the axial PSF. Moreover, in biological samples the imaging depth is restricted by the ability to reject out-of-focus light that otherwise produces a background detrimental to image quality. This is an issue especially for methods with camera-based detection, such as SOFI, but also most SMLM implementations. Optical setup modifications that address this are described in section 3.1.

While newer implementations of STED and SMLM offer superior resolution in 3D, this comes with trade-offs, discussed in detail by Schermelleh and co-workers [45]. STED requires high light intensity to maximally depopulate the excited state, which can lead to photodamage. A similar problem is met when performing NSIM which requires a high laser power in order to achieve saturation. Since STED is a scanning method, high resolution dictates a small scan step and a long acquisition time. Similarly, for SMLM reconstruction a high number of raw data frames is required for one super-resolved frame. As a result, out of methods that are readily applicable to dynamic processes as well as live sample imaging, fluctuation-based techniques are those that offer the highest spatial resolution. Indeed, most of the life sciences applications of SOFI that we list in section 8 use it for live imaging.

Finally, the complexity of a method is another important consideration. From the optical setup point of view, SOFI is very simple to realize: it is sufficient to record a series of frames instead of a single one. This makes it simpler than methods such as SIM and STED that require shaping of the excitation light. On the other hand, like SMLM, SOFI leverages the fluctuations of the sample itself. This means that a fluorophore with blinking behavior on a timescale comparable to the acquisition frame rate must be chosen and labelling density has to be adapted. While these requirements are a limitation, they are less strict for SOFI than those of SMLM, as SOFI can better deal with multiple emitters close together [46].

One of the main limitations of wide-field microscopy is a lack of Z-sectioning (sometimes termed optical sectioning)—the ability to reject out-of-focus light. Consider a three-dimensional, fairly uniform, sample of fluorescent labels under wide-field excitation (figure 4(a)). While only a specific plane is perfectly focused on the camera, each plane in the sample contributes the same amount of light to the image plane. Although the details of these out-of-focus planes are blurred, their high overall intensity overwhelms the faint contribution of the focused features. Thus, samples thicker than a few microns are very difficult to image in this manner.

Zoom In Zoom Out Reset image size

A mathematical formulation of the problem is readily achieved by considering the 3D PSF as a Gaussian beam:

$$\begin{equation} h(r,z)\propto \frac{1}{w(z)^2}e^{-\frac{r^2}{w(z)^2}}, \end{equation} \tag{ 6 }$$

where r and z are the radial and axial coordinates, respectively. The z-dependent beam width, $w(z) = w_0\sqrt{1+\frac{z}{z_R}}$, is defined by the w₀ and z_R parameters, termed the beam waist and Rayleigh range, respectively. If we would image a 3D sample with uniform label density in all directions the contribution of each xy plane would simply reduce to an 2D integral over $h(r,z)$. A reader who is familiar with Gaussian functions can already notice that regardless of the function w(z), integrating over the xy plane yields a constant independent of z. This means that in wide-field microscopy every z-section generates the same integrated intensity, whether it is in the focal plane or quite far from it. Therefore, when imaging a standard biological optically-thick sample with widefield illumination, the signal originating from the focal plane is just a small fraction of the entire observed fluorescence.

Two popular methods that overcome this critical issue are confocal and two-photon microscopy [50, 51]. While the physical mechanism behind these methods vary, from a purely mathematical standpoint, the effective PSF in both is a square of the wide-field PSF (at the wavelength of excitation) [51]; as if the pixels were sensitive only to two simultaneously arriving photons originating from the same emitter. As a result, the signal from out-of-focus emitters, whose emission is spread-out over a large area in the image plane, rapidly decreases with their distance from the focal plane.

Now, if we repeat the reasoning above for a case where the effective PSF is $h^2(r,z)$ (e.g. confocal microscopy and two-photon microscopy), the signal from a plane at a height z is:

$$\begin{align} I(z)\propto \frac{1}{w(z)^4} \int_0^{\infty}e^{-\frac{2r^2}{w(z)^2}}\cdot 2\pi r \,dr \propto \frac{1}{w(z)^2}. \end{align}$$

Here, with growing distance from the focal plane, as the beam expands (w(z) increases), the integrated intensity contribution, I(z), reduces, thus sectioning the imaged sample around the focal plane.

A second-order SOFI image, computed through correlations, also yields an effective squared PSF and therefore provides Z-sectioning that is theoretically as good as in confocal and two-photon microscopy. This property of SOFI has been shown by three-dimensional imaging of a single quantum dot (QD) [22] as well as few-micrometers-thick cells [47] (see figure 2(c)). Yet, practically, Z-sectioning generated through post-processing cannot perform as well as optical sectioning. While the out-of-focus contribution averages to zero a finite-time measurement includes noise which directly translates to unwanted noise in the correlation signal.

Nevertheless, a good example of the inherent 3D potential of SOFI alone can be found in live-cell 3D imaging with a multi-plane wide-field fluorescence microscope [52]. Multi-plane imaging was realized by splitting the emitted light into eight parts and imaging different optical planes onto different regions of two sCMOS sensors. Next, 3D cross-cumulants were calculated for pairs of consecutive depth planes. This approach results in a more homogeneous contrast along the optical axis compared with sequential plane-by-plane 2D cross-cumulants. However, it requires a complicated setup that is sensitive to misalignment as well as carefully selected low-noise detectors.

Imaging of thick samples is a crucial problem for light microscopy in general and for most super-resolution microscopy methods in particular. In that respect, implementing SOFI in conjunction with an optical solution for Z-sectioning is a straightforward way to overcome this challenge.

3.1.1. Total internal reflection fluorescence (TIRF) microscopy

3.1.2. Light-sheet fluorescence microscopy

3.1.3. SOFI in scanning mode

Since super-resolution in SOFI relies on the temporal analyses of collected fluorescence, its principle can be applied together with multiple microscopy geometries. In this section, we discuss the combinations of SOFI with scanning microscopy techniques. The most straightforward example is combining the SOFI with confocal laser scanning microscopy (CLSM). In standard CLSM, the sample is illuminated with a focused laser beam and fluorescence collected through the same lens is focused onto a small pinhole (see figure 5(a)). Since only light from the point conjugate to the pinhole (at the focal plane of the objective) is efficiently transmitted through it, most of the signal from other planes is rejected [58, 59]. CLSM offers optical sectioning with a simple experimental setup.

Zoom In Zoom Out Reset image size

A major disadvantage of CLSM is that acquiring a scanned image, especially in 3D, is time-consuming. To perform SOFI analysis of a CLSM image, one needs to sample the intensity signal at multiple time windows for each single scan step while keeping the entire scan process reasonably fast. For this, a high temporal-resolution detector such as a photomultiplier tube (PMT) or SPAD is required.

Conceptually, the combination of SOFI analysis with a CLSM system provides a robust super-resolution imaging method that maintains the capacity of CLSM to perform 3D imaging [57]. However, due to the point-by-point scanning mode, the acquisition time of such a combination is substantially longer than that of conventional SOFI. As a result, this combination has not yet been experimentally demonstrated.

Image scanning microscopy (ISM), a super-resolved variation to CLSM introduced in recent years, had become a widespread approach to achieve a moderate super-resolution within standard microscopy acquisition times [35, 60]. In ISM, the confocal pinhole is replaced with a detector array containing a few tens of pixels. As schematically shown in figure 5(b), the ISM PSF for each pixel (green) is a multiplication of the laser focus (blue) and the detection probability during the scan (red). This results in a $\sqrt{2}$ narrower PSF that can be translated to a ${\times}2$ resolution enhancement through deconvolution (see section 2.4).

Analyzing the classical [57] or quantum [25] fluctuation contrast in an ISM architecture results in an effective PSF for each detector pair that is even narrower than that of ISM, yielding a ${\times}4$ improvement in resolution [25, 57] for 2nd order SOFI. A comparison of CLSM, ISM and SOFISM (super-resolution optical fluctuation ISM) images analyzed from the same measurement is shown in figure 5(c) [57]. In addition to increased lateral resolution, an improvement in Z-sectioning is achieved in SOFISM. Moreover, reasonable SNR images could be obtained within a short ${\sim}10$ ms exposure time per pixel, far less than the 10's of seconds commonly required for 2nd order SOFI with a camera. This improvement is attributed to the ability of such SPAD detector arrays in sensing fast intensity fluctuations (see section 3.2.2 for an in-depth discussion).

In principle, just like ISM [43, 61], SOFISM can be performed in a multi-beam architecture (for instance in conjunction with a larger SPAD array [62]) and can even be complimented with a two-photon excitation scheme. We expect that such a combination would achieve ${\times}4$ super-resolved images with more than 1 FPS deep within a tissue.

Finally, the combination of camera-based SIM with both quantum [63] and classical [64] fluctuations was suggested as an alternative pathway to wide-field super-resolution beyond the ×2 enhancement of structured illumination. Very recently, the combination of SIM and SOFI has been demonstrated experimentally, achieving up to 2.4-fold image resolution increase for imaging of fixed cells [65].

3.1.4. Spinning disk confocal microscopy

One of the important elements of an imaging system is the camera sensor, making a significant contribution to both the quality and the price of the microscope. Since SOFI relies on image correlations, its requirements from an imaging sensor are somewhat different than those of other super-resolution microscopy methods. The application of currently available commercial microscope camera technologies, EMCCD, CMOS and sCMOS, is discussed in section 3.2.1. A possible future technology for SOFI microscopy, SPAD arrays, is described in section 3.2.2. Specifically, we consider the potential of high temporal resolution in SPADs to enhance the SNR of SOFI.

3.2.1. Established camera technologies—EMCCD, CMOS, and sCMOS

3.2.2. Upcoming technologies—SPAD arrays for imaging fluctuations

Early demonstrations of SOFI used QDs as inherently fluctuating fluorescent labels [22]. QDs are semiconductor nanoparticles whose main advantages in the context of biological light microscopy are a high fluorescent quantum yield, a strong resistance to photo damage (as compared with fluorescent dyes) and a high absorption cross-section [91–93]. Moreover, their narrow emission spectra, tuned by adjusting their size, shape and composition [94], combined with a wide absorption spectra make them highly attractive for multi-color imaging [95].

Interestingly, fluctuations between a bright and a dark state, termed blinking, do not occur over a specific time scale (figures 6(a) and (b)). Initial reports already observed a power-law distribution of dark 'off' periods with longest durations of ∼100 s [96]; recent reports indirectly observed switching even at the microsecond scale [97]. The lack of a specific time scale for fluctuations under specific conditions (in contrast to dyes and FPs) can be problematic for SOFI since it may be unclear what frame rate and number of frames are optimal or even sufficient. However, at relatively high excitation powers (but still below saturation) the power law distribution of the 'off' state is truncated at time-scales of tens to hundreds of milliseconds [98, 99].

Zoom In Zoom Out Reset image size

The underlying mechanism of blinking remains an open scientific question. While some reports considered charging as the cause of dark periods [100], more recent works show that it is unlikely to be a universal explanation for blinking without a definite time scale [101–106]. For a thorough discussion of this topic we refer the reader to comprehensive reviews written on the subject [99, 107].

Currently, antibody conjugated CdSe/CdS core/shell QDs and similar variants are available commercially with a polymeric shell that reduces their toxicity for the sample and improves photostability (Qdot, Thermo Fischer Scientific). However, the generation of such well capped QDs results in relatively sizable objects with diameters of $\gt$10 nm, making it hard for them to penetrate the cell membrane and resulting in non-specific binding within cells [108]. In addition, their varying stability in solutions used for immunochemistry as well as inconsistent penetration depth present challenges for their use in biological research [109, 110]. As a result, while colloidal QDs present a high potential for SOFI and many other microscopy modalities they are still rarely used in such applications.

SOFI with fluorescent carbon nano dots have has also been recently demonstrated [111]. These organic nano particles are more suitable for organic labelling than their inorganic counterparts discussed above. Nonetheless, their relatively long 'off' states hinder the acquisition of higher-order SOFI images.

Fluorescent dyes are popular in bioimaging due to the enormous variability generated by proven synthetic routes, the wide availability of dye-conjugated stains, and the established sample preparation protocols. Dyes are organic molecules that emit light at a wavelength corresponding to the energy gap between ground and excited electronic states. In addition, they can undergo transitions to long-lived dark states that cause fluorescence intermittencies. These processes are strongly influenced by the interaction with the environment.

In a molecule illuminated with visible or ultraviolet light, an absorption transition occurs from the ground singlet electronic state with zero net spin (S₀), to various vibrational levels in the singlet excited state (S₁) (see figure 6(c)). Vibrational relaxation followed by relaxation to the ground state results in the emission of a photon within nanoseconds. Alternatively, an intersystem crossing to the triplet state with non-zero net spin can occur. Due to the spin selection rule, the triplet state relaxation to ground state is dipole-forbidden and therefore has a longer lifetime on the order of microseconds. The transition from the triplet excited state (T₁) back to the ground electronic state is termed phosphorence. Alternatively, the molecules may progress to a second, long-lived dark state D, e.g. through a redox reaction [114] (see figure 6(c)). These states' lifetimes are strongly sensitive to their environment with durations of milliseconds to minutes. Such states are exploited in SMLM methods such as STORM to generate intermittent darkening.

An organic dye molecule in aqueous solution typically emits photons continuously until it is bleached with only short microsecond interruptions when the molecule inhibits a triplet state. Such interruptions are too fast for typical imaging experiments with framerates below 100 Hz. This relatively fast triplet-to-singlet relaxation is caused by dissolved oxygen from air that collides with dye molecules in the triplet state and repopulates their ground state. At the same time, these collisions produce singlet oxygen that is highly reactive and causes unwanted phenomena such as cellular damage [90].

In conventional microscopy, blinking is undesirable and the usual approach is to stabilise fluorescence using reagents that act as triplet state quenchers [90, 115]. On the other hand, for methods relying on stochastic switching, the goal is the opposite: generating dark states with long lifetimes. This is achieved by balancing the concentration of reducing and oxidizing agents, known as the reducing and oxidizing system. A detailed description of the chemical processes regulating photoswitching in dyes is beyond the scope of this review and can be found in [90]. Briefly, the blinking rate of dyes can be tuned by both the illumination intensity and chemical environment [114]. In SMLM methods, most emitters are kept in a long-lived dark state (D) at any given time in order to have less than one bright emitter within a PSF on average [7]. In contrast, fluctuation-based methods can deal with much higher emitter density per frame. For even-order SOFI cumulants, it can be shown that best contrast is achieved for on-to-off times ratio around 1/2 (see section 5.3.2). Note that unlike QD blinking, the switching times of dye molecules are distributed exponentially and a typical time can be mathematically defined [14].

By tuning the buffer composition, photoswitching from dark to bright states can be achieved in many types of dyes. Some of the most frequently used ones are cyanines such as Alexa Fluor 647 which was also the first organic dye demonstrated to be suitable for SOFI [116]. However, photoswitching of other classes of dyes such as rhodamines and oxazines has also been shown [90]. The abundance of photoswitchable dyes enables multicolor super-resolution imaging. In fact, SOFI can be applied to imaging of more channels simultaneously than conventional fluorescence microscopy [117]. By computing spectral cross-cumulants, additional virtual channels are generated and used for subsequent unmixing. Using this approach, simultaneous three-color imaging was demonstrated using only two physical acquisition channels.

Recently a new class of fluorescent dyes appeared, termed self-blinking dyes [118]. Although their blinking, stemming from a reaction called intramolecular spirocyclization, is still influenced by the microenvironment, it has been shown that the on-time ratio in standard buffers is already suitable for SOFI [119].

Fluorescent proteins (FPs) are especially attractive for bioimaging as they can be expressed in cells through genetic manipulation enabling labeling without the complex process of inserting dyes molecules through the cell membrane. Moreover, many FPs exhibit photoswitching whose rate is mostly sensitive to their structure as well as excitation wavelength and not the chemical environment as is the case for fluorescent dyes. A detailed discussion of the mechanism of this switching is beyond the scope of this paper and can be found, for example, in an extensive review of the photoinduced chemistry of FPs published a few years ago [120].

One of the best known reversibly photoswitchable proteins is Dronpa [121, 122], one of several green FPs derived from corals. Photoswitching from the dark back to the fluorescent state is achieved by illumination with UV light, although excitation of both fluorescent and dark state with one laser wavelength is possible (see figure 6(d)). Typical blinking dynamics of Dronpa under 488 nm excitation shows an exponential distribution of lifetimes with time constants in the range of tens of ms, with the exact value depending on the excitation power [121]. This means that the frame rate required to observe Dronpa blinking is well within the range of modern camera-based microscopes.

Other popular FPs that exhibit photoswitching are green fluorescent mEOS2 derivatives mEOS3.2, SkylanS and SkylanNS as well as red fluorescent mCherry. Of those, SkylanS [123] was developed specifically for SOFI. A systematic comparison of performance of 20 photoswitchable FPs at different imaging conditions along with the raw data can be found in [88, 124]. Out of the tested FPs, blue and cyan ones performed poorly, while green and red ones enabled a good SNR in second-order SOFI (in some cases with an additional UV illumination to assist with photoswitching). Notably, it has been shown that the green- and red-fluorescent photoswitchable proteins can be used together to achieve two-color or photochromic SOFI (pcSOFI) imaging [125].

The usability of organic dyes and FPs for SOFI is limited by their inherent susceptibility to photobleaching. Glogger and coworkers presented an approach that circumvents this issue by using reversibly and transiently binding labels [126]. Based on PAINT and specifically DNA-PAINT [13], the method relies on fluctuations arising from attachment and detachment of fluorophore-labeled short DNA oligonucleotides to specific binding sites. These are labeled with target-specific antibodies containing complementary DNA structures. The probes are continuosly replenished from a buffer which serves as a reservoir, giving the method a highly sought-after immunity to photo bleaching. These labels are available commercially and have been used for SMLM as well as STED. Combining them with SOFI enables using shorter acquisition times through denser frames.

While the formulas given in section 2 are continuous in space and time, an image series acquired with a camera-based microscope consists of discrete pixels and frame acquisitions. The typical raw SOFI data consists of an image series of several hundreds or thousands of frames, reconstructed to obtain a single super-resolved frame.

For auto cumulants, processing a pixelated image series is straightforward: auto cumulants are calculated for each pixel independently with a time lag τ that takes integer values corresponding to a difference in frame number. Notably, since auto-correlations for $\tau$ = 0 contain contributions from noise sources with short correlation times such as shot noise, they should be used with great caution for image reconstruction. Similarly, for higher-order auto-cumulants, the delays $\tau_1, \tau_2, {\ldots}, \tau_{n-1}$ should be all different.

As evident from equations (2) and (3), any combination of non-zero and non-repeating delays (and any linear combination of such cumulants) would generate a super-resolved image with the same resolution. In general, a typical correlation function of blinking emitters decreases quickly with delay (see figure 7(a)), so selecting the shortest possible delay times leads to the highest level of correlation signal [127]. For long time delays, the effect of bleaching becomes apparent. Although at a first glance bleaching yields strong, noise-free SOFI images, it has been shown that the result does not correspond to the actual structure of the sample. Instead, cumulants computed for long time delays can be used to estimate the contribution of bleaching and if necessary, correct for it [89].

Zoom In Zoom Out Reset image size

Although theoretically auto-cumulants offer the same resolution improvement as cross-cumulants, in practice, the resolution of an optical microscope is limited not only by PSF, but also by the camera pixel size. Therefore, it can be useful to calculate cross-cumulants in order to generate additional virtual pixels. An additional advantage is that by omitting the correlation of a pixel with itself, the above-mentioned effect of temporally-uncorrelated noise is avoided so that cross-cumulant can be computed for $\tau_k = 0$, yielding higher SNR and simplifying the computation [41].

The principle of creating a virtual pixel by computing second-order cross-cumulants is shown in figure 7(c). To obtain a new pixel, pairs of opposite pixels are cross-correlated. It is apparent from the 'distance factor' in equation (4) that it only makes sense to include pairs up to one PSF width apart. Moreover, the distance factor serves as a weight for the contributions from pixel pairs with different distances and has to be divided by when combining different cross- and auto-cumulants into a single image, to avoid the so-called checkerboard pattern. If the PSF is known, calculating the distance factor is straightforward. Alternatively, considering the cross-cumulant of pixel pairs with different distances and assuming a Gaussian PSF, one can estimate the PSF width and use it for subsequent image reconstruction (see figure 7(b)) [41, 124]. The theoretical formalism for this analysis is outlined in [128]. Recently, a model-free approach to correct pixelation effect has also been demonstrated [129].

For higher-order cross cumulants the principle of computing virtual pixels is similar. The fluorescence signal from a set of n pixels can be used to create a new pixel in their geometrical center. However, one needs to keep in mind that an nth order cumulant contains contributions from correlations of all orders from 2 to n. The procedure of computing a cross-cumulant for a set of n pixels from correlations of its subsets is explained in detail in reference [130].

Finally, in a practical implementation of SOFI one must consider that correlations can occur from more banal sources than fluorescence fluctuations. A small drift or vibrations in the position of the stage, for example, will generate a signal that is proportional to the spatial derivative of the sample structure. Fluctuations in the illumination source power will also cause unwanted correlations. It is important to note that unlike correlation in emitters' fluorescence, such a signal would seldom contain any extra information that can be applied to super-resolution and is more likely to distort the SOFI image.

The theoretical concepts on which SOFI is based assume an infinite recording time. To obtain an image of a sample containing blinking emitters, a sufficiently long recording is required to for a trustworthy estimation of the average intensity of the emitters. Similarly, the cumulants require a sufficient frame number to converge toward their theoretical values.

The number of frames required for faithful image reconstruction depends on emitter statistics, that is, their blinking rate as compared to the acquisition frame rate and on-time ratio, emitter density and the noise level [46, 127, 131, 132]. Under good imaging conditions, second-order cumulants converge within a few hundred frames, corresponding to a few-seconds acquisition with a fast camera. For higher orders, this number grows quickly to a few thousand for the third and more than 10 000 frames required for the fourth order [127]. These order-of-magnitude estimates were obtained using numerical simulations and depend on the assumed imaging conditions and the exact data analysis procedure used for SOFI. The dependence of acquisition time on the order of cumulant is a complicated topic and it is difficult to provide clear and general guidelines. This issue is further complicated in a real experimental setting due to fluorescence bleaching, sample drift and similar factors. Various ways for noise filtering of the raw image stack have been proposed to decrease the frame number [133] or increase the cumulant order that can be computed for a given frame number [134], but have not been widely applied yet. Data pre-processing can also be used to correct for photobleaching artifacts and as a result improve the contrast of SOFI images [135].

In conventional microscopy, longer exposure times and higher illumination intensities yield a higher SNR. At the same time, for SOFI, the frame exposure duration has to be short enough to register blinking. The optimal exposure time will be a trade-off between these two factors. Similarly, illumination intensity should be chosen so that it does not cause bleaching or damage the sample during the long recording.

5.3.1. Non-linearity and balanced SOFI

5.3.2. Cusp artifacts

Another issue that limits the use of higher cumulant orders is the sign dependence of cumulants, of order three and higher, on the emitter blinking statistics. This was first described in detail by Yi and Weiss [137] and termed a cusp artifact.

The time that an emitter spends in bright and dark states depends on emitter's properties and its environment, including factors such as local chemical environment, emitter orientation, proximity to other fluorophores or local excitation power. Moreover, all fluorophores experience bleaching which is equivalent to permanently staying in a dark state. The impact of the switching statistics on the SOFI image can be understood by looking at equation (3) and assuming that an emitter switches only between two states, 'on' and 'off'. Then, if ρ is the on-time ratio, the emitter's nth order cumulant is proportional to an nth degree polynomial $\omega_n(\rho)$ [137]. The first five ω_n polynomials are shown in figure 8(a). Naturally, all polynomials reach zero for ρ equal to 0 or 1, corresponding to an emitter permanently remaining in the same state, i.e. not blinking. Importantly, while the second-order cumulant is always positive, higher-order cumulants take both positive and negative values. If the value of ρ could be well controlled for all emitters this would only result in a constant multiplication factor. However, given the number of factors influencing the fluctuation statistics, this is typically difficult to achieve. As a result, the varying contrast sign can severely distort the SOFI image.

Zoom In Zoom Out Reset image size

Simplified simulations shown in figures 8(b)–(e) illustrate how the change of cumulant sign leads to artifacts. A simple simulated image of three emitters is shown in figures 8(b)–(d). First, emitters with identical properties present an artifact-free image (figure 8(b)). In the next panel, we assign a negative sign cumulant to one of the emitters resulting in dark boundaries between the positive and negative areas, which are the origin of the term 'cusp'. In the bottom panel, the emitters have different cumulant amplitudes, as well. Importantly, each such combination leads to a substantially different image, showing how difficult it is to interpret higher order SOFI images. Finally, figure 8(e) adapted from [137] shows a more realistic simulation that includes multiple emitters as well as noise and bleaching. As soon as a low amount of noise is introduced, in combination with the cusp artifact, it degrades higher order cumulants to such an extent that no image can be reconstructed beyond the second order.

In another paper Yi and co-workers propose a solution to circumvent cusp artifacts [130]. Their approach is to reconstruct the image using statistical moments instead of cumulants. In contrast to cumulants, even order moments do not exhibit sign change. However, this solution entails a problem, since moments of order higher than 2, contrary to cumulants, do not eliminate cross-terms coming from multiple emitters and therefore do not, as such, produce a super-resolved image (see section 2.2.1). As a result, the cross-terms limit the resolution enhancement to a factor of $\sqrt{2}$. To overcome this issue, deconvolution algorithms can be applied to even-order moments. The authors claim that this enhances the resolution further, leading to a total improvement by up to a factor of $\sqrt{2n}$ (compared to n in the case of deconvolved SOFI). In the paper, super-resolution image reconstruction from live-cell-imaging data up to the 6th order is shown and compares favorably to other methods, including bSOFI.

While the lowest order auto-cumulant is straightforward to compute in any programming language, the implementation of higher order auto- and especially cross-cumulants is much more involving. Furthermore, increasing the cumulant order also increases the number of pixel and time lag combinations to include in the computation. This means that carefully written code using efficient dedicated libraries becomes more important. A good starting point is to use existing packages which will be briefly described here.

Localizer is a package for super-resolution microscopy image analysis created by Dedecker and co-workers [138]. It offers the option to calculate SOFI cross-cumulants of orders up to 6 with many available options regarding, for example, the number of pixel combinations included (where the default is calculating the cumulant only between pairs of closest pixels). Localizer is written in C++. A GUI version is provided in the form of a plugin for Igor Pro (WaveMetrics, Inc.). A Matlab MEX file is also available.

Recently, another Igor Pro plugin that based on Localizer has been published. SOFIevaluator [88] is a tool that analyses a raw image by calculating a set of metrics describing the quality of data for the purpose of SOFI analysis, such as the SOFI SNR and the magnitude of bleaching. It can be used to quantitatively compare different fluorophores and parameters for image acquisition in order to select the best imaging conditions for a given experiment.

For bSOFI [136], a MATLAB toolbox has been developed for two and three-dimensional SOFI analysis [139]. It allows to reconstruct the image using third and fourth-order cumulants.

A simulation tool has been developed to test the influence of different factors on the outcome of super-resolution reconstruction [140]. The package includes simulation of SOFI cross-cumulants of different orders, as well as bSOFI. Moreover, it allows to compare SOFI reconstruction to STORM. Its purspose is to get an idea of what imaging conditions would lead to a successful analysis before running an experiment. It can be also a learning tool, since the processing steps of SOFI are visualized and explained in a tutorial-like way.

The initial breakthrough demonstrations of SMLM used sparse frames with well-separated emitters, thus avoiding the risk of overlapping PSFs [6, 7]. The result was a trade-off between superb resolution and a prohibitively long acquisition time of a few hours. Even though improvements in emitter brightness and switching times reduced the total exposure time, the low density of each frame was the obvious main hurdle for live-cell imaging. As a result, within a few years initial attempts to algorithmically analyze images of overlapping emitters emerged [73, 141, 142]. While single emitter localization methods failed to efficiently process raw frames with >1 μm⁻² emitter densities, methods such as DAOSTORM and CSSTORM where able to analyze fairly well frame densities of up to 5–10 µm⁻² [141, 142]. Accordingly, the acquisition time was reduced by an order of magnitude to a few seconds per super-resolved frame [143].

From an algorithmic point of view we can divide these methods into two groups. In the first, an iterative process locates bright emitters (and groups) and removes them from the image in order to find the residual emitter locations [141]. The success of such an approach critically depends on precisely characterizing and modeling of the experimental PSF [73]. In addition, such multi-emitter localization procedures are computationally demanding.

The second group of algorithms is based on compressed sensing. Generally, a reconstruction algorithm attempts to minimize a positive-valued 'cost function' which approaches zero when the fit matches the experimental data. A common example of such a function is the sum of squared differences between image and model (squared errors) over all pixels. Whereas the previous algorithm group attempted to minimize only the sum of squared errors, a compressed sensing algorithm also imposes sparsity. That is, the cost function increases with the use of more emitters to reconstruct the image. This approach avoids over-fitting of the noise patterns with many emitters and applies sparsity of frames as extra information.

Both algorithmic families have developed over the past decade and already include many sub-methods and software packages available for users. A comparison of their performance is an incredibly intricate task which is tackled for example by the Super-resolution fight-club competition [20, 21]. The quality of analysis method can be measured by multiple metrics (localization rate, false positives, false negatives, localization accuracy etc) and will depend on the emitter density, SNR and sample structure (2D vs. 3D, lines vs. areas etc) in a specific experiment.

Finally, it is instructive to note that these methods are developed with the conditions of SMLM experiments in mind. Thus, they typically include only a minimal consideration for the time domain and treat subsequent frames as independent. An exception are methods that identify emitters that appear in multiple consecutive frames and average their localization position (temporal grouping) [21, 144]. However, a recorded image series of fluctuating emitters contains much more information in the temporal domain that localization reconstructions tend to overlook. For example, emitters are typically active in a few different periods along the acquisition time. The following section introduces computational microscopy methods that make use of both the temporal and the spatial information in the raw data. With this combination they aim to reconstruct frames with a higher density of emitters and thus provide super-resolved images at a higher frame rate.

Whereas the above-mentioned computational imaging techniques were conceived with the raw data of SMLM in mind, recently a lot of attention was devoted to methods for reconstructing super-resolved images from SOFI-like data sets. That is, analyze dense frames in which emitters rapidly switch back and forth between the bright and dark states. Containing far more emitters, such dense frames have the potential to produce a faster super-resolved reconstruction of a sample. We emphasize here that the above-mentioned change from SMLM to SOFI-like data requires a conceptual shift. In the latter, information about the sample structure is stochastically spread over the time series and extracting it is a convoluted task.

Before we approach methods that, like the original SOFI analysis, make direct use of auto- and cross-correlations, we describe two techniques that address fluctuations differently. The first, multiple signal classification algorithm (MUSICAL) uses singular value decomposition (SVD) to decompose an image stack to orthogonal components [145]. After reshaping the image stack into a 2D matrix with one dimension for time and the other for all pixels, SVD decomposes the stack into a sum of components, each of which is a product of a temporal and spatial vector. A weight factor determines their level of contribution to the image stack intensity. One can then attempt to separate signal from noise components and calculate a so-called indicator function pointing to the location of the emitters.

A second strategy to decompose the fluorescence fluctuation image stack is adopted by the Haar wavelet kernel (HAWK) analysis [18]. Data from emitters that partially overlap in time and space, are separated by considering the length and frequency of blinking events. This generates a substantially longer data set, increasing the level of sparsity in the data. Applying an additional algorithm, such as multi-emitter localization produces the super-resolved image. In fact, a SOFI or b-SOFI analysis combined with a HAWK pre-processing can be advantageous in avoiding non-uniform illumination artifacts [146].

A few computational imaging methods come even closer to the original concept of SOFI, incorporating temporal correlations. A notable example is sparsity-based super-resolution correlation microscopy (SPARCOM) [23]. In SPARCOM, the concept of signal sparsity is applied to the pair-wise correlation of all pixels in the image. Assuming that emitters fluctuate independently, the correlation signal is even more sparse than the original image stack, with non-zero values centered only around the diagonal. As a result, a sparse reconstruction of the signal can perform well up to higher frame densities. According to simulations SPARCOM analyzes scenes with >10 μm⁻² emitter density with less than $10\%$ false positive localizations. Similar approaches were recently used to reconstruct images from the quantum [147] and fluctuations where images of Q-ISM and ISM were merged using a sparsity reconstruction to generate super-resolved images at a few millisecond per pixel exposure. Following this line, a recent report from the same groups performed reconstruction of SOFISM data sets (see section 3.1.3), showing that reconstructing the entire scan image stack can outperform pixel reassignment in terms of resolution [148].

A non-optimization approach to correlations signal is taken in super-resolution radial fluctuation (SRRF) microscopy [149]. In SRRF, each frame is transformed into a radiality map, a non-linear transformation which heightens sample features in which the intensity gradient point to a an origin. A temporal correlation of a radiality stack yields a super-resolved reconstruction for densities in which multi-emitter localization algorithms fail. A promising demonstration of SRRF showed fixed-cell super-resolved images at a 1 Hz frame rate [150]. A recent comprehensive comparative study, however, found substantial discrepancies between SRRF reconstructions and the ground truth object, especially in non line-like features [19], while another study by Yi et. al have found that SRRF achieved a very high resolution at the expense of distorting some features and in particular artificially narrowing line features. We note that while SRRF generate a clever non-linear transformation of the SOFI data, it does not provide any extra information temporally or spatially and is therefore not expected to perform well as other method described in this section.

Finally, we review the method of entropy-based super-resolution imaging (ESI) [151]. Its analysis requires the computation of Shannon information weighted temporal-correlations. Although the correlation orders are differently classified in ESI it seems that it produces a similar resolution to the original SOFI analysis.

In recent years, deep learning techniques have been applied to the most challenging tasks in science and engineering including the fundamental limits of optical microscopy [152–154]. The hope of advancing far-field super-resolution microscopy drove a lot of attention to the application of deep learning techniques to analyze the standard data sets measured for SMLM and SOFI [24, 155, 156]. In this sub-section we provide a few examples of such attempts and briefly discuss the potential and risks involved in deep learning for super-resolution microscopy.

Briefly, a neural network algorithm can be thought of as multiple layers in which a linear manipulation of the raw data is followed by a non-linear step that zeros lower values. The parameters for the linear operations in each layer are optimized in the training phase where a ground truth dataset is compared with the output of the network. After training, the network is ready to tackle a similar analysis for previously unseen data.

In a similar spirit to the works presented in section 6, the basic challenge in the works described below is achieving SMLM-level super-resolution within more reasonable acquisition times. One approach, termed ANNA-PALM, attempts to do so by training a network to recognize structures according to under-sampled SMLM data [155]. The network is trained to reconstruct an super-resolved SMLM image, generated from a large number of low-density frames, from a small subset of frames. Impressively, for simulated data, the network achieves a similar quality of reconstruction as SMLM for roughly ${\times}26$ less frames. Experimental images of micro-tubules within fixed-cells, that agree with the SMLM ground-truth, were obtained from just tens of frames with an overall exposure time of $\lt$1 s. However, such an approach is very specific to the type of sample for which the network is trained. In essence, in the lack of sufficient localization to sample a super-resolved image (sampling below the Nyquist rate), the network completes the image with patterns that are most similar to those that it was trained on.

To avoid such a risky sensitivity to sample type caused by under-sampling, Deep-STORM attempts to improve the analysis of dense frames containing plenty of information [24]. Here, the network simply tries to outperform the previous multi-emitter localization algorithms in finding positions of overlapping emitters. Training with images of sparse in-vitro blinking QDs, the authors successfully reconstruct images of micro-tubules labeled with the same QDs. In the context of this work, we note that one common advantage of neural network solutions is the ability to reconstruct images without user-tuned parameters, a process that is time consuming and requires a lot of expertise. In a more recent work from the same groups, the authors show very good results in reconstructing 3D SMLM data [157]. They combine deep learning with a PSF engineering approach: the 3D PSF of the microscope is adjusted to a so-called Tetrapod, shown to contain a higher level of information on the 3D location of a point emitter [9]. Analyzing the images with such a PSF, a convolutional network obtains 3D super-resolution images even in relatively high emitter densities. An exciting approach examined in this work is the optimization of the optical system itself (PSF engineering) together with the encoding deep neural network. Through simulations, the network 'discovers' a new optimal PSF that is better behaved than the result of a previous numerical optimization, the Tetrapod function, in high-density localization.

The final approach we review here is termed LSPARCOM and is based on the classical sparsity-based SPARCOM algorithm, introduced in section 6.2. This network shows two advantages over the two previous implementations: its is less sensitive to experimental parameters such as the sample structure redundancies (like ANNA-PALM) or the imaging parameters (like DeepSTORM) and it uses a less deep and more interpretable network [156]. In fact, the authors claim that the first property is the result of the second; using a small number of training parameters does not allow the network to learn training-set specific feature of either the sample or the optical setup. Such flexibility can be very helpful for experimental applications where the PSF itself, for example, can depend on the sample through scattering and refractive index.

Naturally, when observing the results of neural networks in super-resolution a word of caution is in place. Like for other algorithms, it is difficult to ascertain that the reconstruction is indeed trustworthy; however, as so-eloquently phrased in [154] deep networks are 'specifically trained to lie plausibly'. That is, they are trained to use among others the structural redundancies in their training sets in order to extend the amount of information stored in the data. It is therefore re-assuring to note that DeepSTORM can be trained on randomly positioned emitters and performs very well on cell-like simulations and experimental biological samples. LSPARCOM presented in-part such flexibility by training on different types of simulations, but it seems that all training and data samples included line feature with intersections.

While the description of the different algorithmic approaches in this section focused mainly on their potential for super-resolution microscopy, we note that valid concern is often raised with regard to the use of reconstruction algorithms. Unlike most of the early demonstrations of super-resolution microscopy (e.g. SMLM, STED, SOFI, SIM and ISM) algorithmic reconstruction, by nature of being an optimization problem, is not based on a closed-form mathematical model. When such a model is not present we cannot be assured that a super-resolved image truly represents the object, even under proper experimental conditions. Whereas, for example, a SOFI image under poor experimental condition appears naturally noisy and difficult to interpret, an algorithmic reconstruction may appear completely clear with features that do not reflect the structure of the object whatsoever.

Since most of the methods described above have been introduced within the past 5 years, it is certainly too early to firmly state in which situation a specific method is useful. As external observers, we believe that an impartial testing grounds for different algorithms under various conditions (simulated and experimental), akin to the super-resolution fight-club [21], is critically important for the progress of this field. Although many of the works cited here follow the high standard of comparing themselves to multiple other algorithms, we feel that the level of expertise in operating these complex methods is not equal and generates bias in such comparisons.

Intensity correlation measurements of light, now the cornerstone of quantum optics [160, 161], were first investigated in the context of astronomy [162]. In their seminal experiment, Hanbury-Brown and Twiss analyzed a setup in which light is split at a beam splitter and falls onto two photodetectors [163]. This setup is schematically represented in figure 10. The intensities measured by the two photodetectors are correlated. The intensity correlation is typically plotted as a function of time delay (see the bottom part of figure 10). For thermal light sources, intensities measured by the two detectors are positively correlated at small temporal delays as a consequence of the fluctuations of the emitted light intensity. This tendency of photons from a fluctuating source to be detected together is called bunching. Single photon emitters, in turn, exhibit an intensity anti-correlation at small delays, a phenomenon dubbed antibunching (see figure 10(a)). This is a consequence of the fact that a single emitter can emit at most one photon at a time. Antibunching, observed for the first time in the fluorescence of atoms in 1977 [164], is one of the first experiments that can only be explained in terms of a quantum theory of light [165]; it can also be considered as the first experimental proof for the existence of photons.

Zoom In Zoom Out Reset image size

The idea to use photon pair detection in a fluorescence microscope to achieve super-resolution has been introduced by Hell et al [166]. It remained impractical because of the lack of fluorescent two-photon emitters until Schwartz and Oron noticed that every single photon emitter can be considered a source of missing photon pairs [159] (the idea explained in more detail below). Soon after their proposal, the idea has been experimentally realized with QDs [3] and Nitrogen-Vacancy centers [167]. In fact, most emitters used in fluorescence microscopy, including fluorescent dyes [168] and proteins [169] as well as QDs [170], are single-photon sources. As a result, they can be used as labels for antibunching microscopy.

Antibunching is a single-emitter phenomenon in the sense that a single-photon emitter, such as a dye molecule, can only emit one photon after it is excited and then needs to be re-excited before it emits again. This does not mean, however, that antibunching cannot be observed when more than one emitter is present in the observation region. In such a case there are still less photon pairs emitted simultaneously than at a non-zero delay, and we observe a dip, though not a zero value, of the second-order correlation function at zero delay. The area of this dip, representing the number of missing photon pairs, is the antibunching microscopy contrast.

Antibunching-based imaging was first demonstrated in a wide-field microscope which required an ingenuous experimental setup with pulsed excitation frequency-matched to the frame rate of an EMCCD camera. Working at the kilohertz rate resulted in long acquisition times, limiting the practical applicability of the method [3]. The other demonstration that used nitrogen-vacancy centers [167] was performed in a confocal setup with fast detectors. NV centers offer excellent photo-stability, but are rarely used for fluorescent bioimaging. These approaches had to be further developed to go beyond a proof-of-concept demonstrations.

Recently, quantum antibunching microscopy has been combined with another super-resolution microscopy technique, namely ISM [58–60] (introduced in section 3.1.3). This combination is termed Q-ISM [25]. This followed a theoretical proposal for a wide-field variant, combining quantum correlations and SIM, by Classen et al [63]. They found that, theoretically, the resolution improvement factor surpasses the one obtained by each method separately. Q-ISM was used to obtain super-resolved images of a microtubule sample labeled with QDs. Figure 11 compares performance of Q-ISM to that of CLSM and classical ISM on the same data set collected by an ultrafast camera composed of a fiber bundle connected to single photon avalanche diodes (see subsection 3.1.3). A CLSM image (figure 11(a)) is obtained by adding the count rates collected by different detectors at each step of the scan, whereas for ISM (figures 11(b) and (c)), images acquired by the individual detectors are shifted prior to addition.

Zoom In Zoom Out Reset image size

It can be observed in figure 11 that Q-ISM features higher resolution than ISM. Unfortunately, because of losses and non-unity QEs of fluorescence emission and detection, the detected number of missing photon pairs is substantially smaller than that of overall photons. As a result, Q-ISM suffers from a smaller SNR compared to ISM. One should keep in mind that the Q-ISM signal is recorded together with the ISM signal and therefore it seems natural to combine the information present in the two signals. This can be achieved algorithmically, for example by using joint sparse recovery methods as demonstrated in [147].

First demonstrations of antibunching microscopy relied on avalanche photodiodes for confocal modality and EMCCD cameras for wide-field modality. EMCCD cameras offer a high number of pixels and an excellent QE, but are slow (see section 3.2.1). In [171] an ultrafast 'camera' was built using a fiber bundle, multiple SPADs, and a time-tagging unit. Each pixel in this camera was capable of timing detection at sub-nanosecond resolution, but the number of pixels was quite limited. The emerging technology of SPAD arrays offers a promising avenue for antibunching microscopy in confocal [86] and wide-field modalities (see section 3.2.2). Antibunching microscopy can be considered to be a quantum analog of SOFI and as a result can be described by an analogous theory leading to the same PSF functions as standard SOFI. Nevertheless, the physical phenomena underlying the correlations are entirely different. In the case of standard SOFI the correlations result from emitter blinking, whereas antibunching, a genuinely quantum phenomenon which marks the reduction of fluctuations (sub-Poissonian statistics) is the source of correlations in quantum SOFI (see figure 10). This fundamental difference between the two techniques bears practical implications that we will describe in the following section.

Blinking statistics and dynamics depend heavily on the chemical micro-environment and the excitation power (see section 4) and as a consequence the contrast of SOFI microscopy carries additional information about these factors, but at the same time it is prone to artifacts. In contrast to blinking, antibunching is independent of the above mentioned factors and used as a microscopy contrast is less prone to artifacts. Many fluctuations can result in positive correlations which will generate an artificial contrast e.g. laser power fluctuations or stage position oscillations, but it is much less common to generate false negative correlations through technical imperfections.

The time scales of blinking and antibunching are different. Antibunching is governed by the fluorescence lifetime which lies in the nanosecond range, whereas blinking typically occurs at the timescale of microseconds up to seconds. For continuous excitation, the requirement for antibunching microscopy is to resolve detections at a resolution below the fluorescence lifetime, which is technically challenging. This issue can be circumvented with the use of pulsed excitation, as long the pulse duration is much shorter than the fluorescence lifetime [3, 25]. While there are exceptions, typically photon counting detectors (Geiger mode), are used in quantum correlation SOFI [86], whereas the implementation of classical SOFI is simpler since it can be conveniently measured with slower detectors such as standard cameras.

Measurements of quantum correlations can be also used to count emitters within a diffraction-limited spot of a confocal laser scanning fluorescence microscope [172–176]. This is analogous to bSOFI (described in section 5.3) which can extract the information about emitter properties from a combination of different correlation orders [136]. It is quite intuitive that the antibunching contrast carries information about the number of emitters within the observed spot. If the normalized second order correlation function, $g^{(2)}(\tau = 0)$, is close to zero, we infer that we are observing a single emitter, for a high number of emitters $g^{(2)}(\tau = 0)$ approaches 1. Generally,

$$\begin{equation} g^{(2)}(\tau = 0) = 1-\frac{1}{M}, \end{equation} \tag{ 8 }$$

where M is the number of active emitters within the observed spot with an equal intensity of emission. As can be seen from expression (8), emitter counting using the value of $g^{(2)}(\tau = 0)$ can work reliably for a small number of emitters. Recording multi-photon statistics (higher order correlations) enables inferring the emitter number with precision of a few percent even for tens of emitters [172]. Studies performed in the Herten group demonstrated the feasibility of emitter counting based on photon statistics for various fluorescent dyes [173]. A systematic comparison of performance of dyes in this technique is given in [176]. In [177] molecule counting by photon statistics has been implemented within a confocal microscope. This enabled the authors to obtain maps containing the number of molecules contained in sub-diffraction regions.

Emitter counting by photon statistics can be used to enhance super-resolution microscopy. It has been combined with STED microscopy [177] and used to speed up data acquisition in a proof-of-concept localization microscopy measurement [171]. Israel et al have demonstrated that $g^{(2)}$ measurements can be used to aid fast localization microscopy [171]. The speed of localization microscopy is typically limited by the requirement for emitters not to overlap spatially within any frame (see section 6). The information about the number of active emitters within the diffraction limited spot obtained from $g^{(2)}(\tau = 0)$ measurements allowed Israel et al to filter-out the unwanted frames with multiple active emitters.

Although SOFI for bioimaging was first demonstrated more than 10 years ago [22], it remains significantly less popular than SMLM, most likely because of the modest resolution improvement it typically offers. However, the need to reduce acquisition time and illumination intensity in live-cell imaging makes it challenging to use SMLM. SOFI performs better than SMLM at low SNRs and is applicable over a wider range of labeling densities and blinking rates [46]. The latter also means that it can be used without the specialized buffers used to control fluorescent dyes blinking rates that are usually incompatible with live-cell imaging.

Fluctuation-based imaging is well-suited to time-lapse microscopy in live cells, where a long measurement, lasting minutes or even hours, is performed to obtain a movie with frames that are seconds or tens of seconds apart. For instance, Sirianni and co-workers used SOFI for two-color imaging of mitochondrial fission [178]. A TIRF microscope was used to acquire 100 frames in each of the two channels every 30 s for several minutes. A second-order SOFI image was generated from each 100 frames, forming a two-color super-resolved time-lapse movie.

Even when no time-lapse recording is taken, live-cell imaging, while more technically demanding, is sometimes preferred. Cell fixation can alter its structure, meaning that the tissue might look different than in vivo. In this case it might be required to perform live-cell imaging to verify that the sample preparation itself does not change the morphology of the sample. For example, Scholefield and coworkers [179] used fourth-order SOFI to verify that the lattices formed by the NEMO protein observed using STORM in fixed cells also appear in living cells.

Apart from increasing the image resolution without the disadvantage of non-linear dynamic range expansion, bSOFI [136] allows to extract quantitative maps of parameters such as molecular state lifetimes, concentration and brightness distributions within the samples. As such, it can be used for quantitative fluorescence microscopy, for instance in situ protein counting, offering reasonably good temporal resolution of the order of 10 s and lower phototoxicity than some other methods used in this field [180]. bSOFI was used to measure the density of the protein paxilin in focal adhesions of mouse embryonic cells [181, 182] and was used to study the nanoscale distribution and clustering of another protein, CD4 mutants, in the plasma membrane of T cells [181].

The examples described above use fluorescence blinking as a means for obtaining a super-resolved image. However, blinking itself can be used to characterize properties of the sample. Outside life sciences, this has been applied to imaging of inhomogeneities in perovskites, resulting in different dynamics of photoluminescence blinking [183]. Since SOFI is based on calculating correlations of the signal fluctuations, it could be used to establish the size and to some extent the number of separate emitting regions in the investigated areas of the crystal. By comparing this to results of scanning electron microscopy, the authors proposed a model for how inhomogeneities of different sizes influence photoluminescence quenching and blinking, which in turn influences the performance of perovskite films in solar cells and light-emitting devices.

Optical fluctuations imaging has been used to develop a new class of fluorescent biosensors, FLINC [26]. Many existing biosensors are based on the idea that a change of distance between two molecules, such as binding or cleavage of a bond, can be detected by attaching fluorophores to both of them. The most common way to detect the change is to detect energy transfer between the two fluorophores (Förster or fluorescence resonance energy transfer, FRET) by measuring the fluorescence spectra or lifetime. The mechanism used in FLINC is somewhat different. Here, the proximity of two FPs changes the fluorescence fluctuation behavior of the readout protein. It can be seen as an extension of the work of Cho and co-workers [184] where blinking induced by a FRET process was used to increase the image resolution. In the work of Mo and co-workers, the two interacting FPs, Dronpa and Tag-RFP-T, are fused to two different ends of a proximity-dependent reporter or to two different target molecules. In both cases, the biosensor activation or target molecules fusion brings the two FPs together. The proximity of Dronpa dramatically increases the fluorescence fluctuations of Tag-RFP-T. FLINC-based biosensors are fast and reversible and since they are analyzed by computing SOFI contrast, they offer intrinsically increased resolution.