Advances in Mask-Modulated Lensless Imaging

Abstract

Lensless imaging allows for designing imaging systems that are free from the constraints of traditional imaging architectures. As a broadly investigated technique, mask-modulated lensless imaging encodes light signals via a mask plate integrated with the image sensor, which is more compacted, with scalability and compressive imaging abilities. Here, we review the latest advancements in mask-modulated lensless imaging, lensless image reconstruction algorithms, related techniques, and future directions and applications.

1. Introduction

Unlike conventional focusing-based imaging, lensless imaging utilizes optical encoders for scene capture. Images captured by image sensors undergo modulation and are often not directly usable. Instead, they are decoded using compatible image reconstruction algorithms to reconstruct the scene [1]. Depending on the different modulation mechanisms in the imaging chain, lensless imaging can be divided into two main categories: illumination modulation and mask modulation. Lensless imaging systems based on illumination modulation utilize characteristics such as the position, coherence, and pulse timing of the illumination source to capture images with different encoded illumination. Depending on the specific illumination mode, lensless imaging can be further categorized into projection imaging, holographic imaging, and time-resolved imaging.

Mask-modulated lensless imaging achieves encoding modulation of the target scene by placing optical masks in front of the image sensor. Furthermore, with designed masks, functions such as 3D imaging, depth estimation, and spectral imaging can be realized. An amplitude mask encodes the incident light by partially blocking or attenuating it, as seen in methods like pinhole imaging and coded aperture imaging. Phase mask modulation, on the other hand, utilizes the principles of physical optics to modulate the phase of light. It almost does not block or attenuate light, effectively addressing the issue of lower light flux in amplitude mask modulation and improving the image signal-to-noise ratio. Common phase mask types include phase gratings, diffusers, and diffractive optical elements (DOEs).

In this review, an overview of the recent developments in lensless imaging using mask modulation techniques is provided in Section 2. In Section 3, the basic principles of image reconstruction, especially compressive imaging and deep learning algorithms, are explained. In Section 4, the related techniques and future directions are provided. Section 4.1 explores deep optics methods for optimizing both mask design and reconstruction algorithms at the same time, Section 4.2 describes the diffractive and optoelectronic neural networks for reducing energy consumption, and Section 4.3 gives a brief introduction on single-pixel imaging. Section 5 and Section 6 discuss the applications and prospects, respectively.

2. Advances of Mask-Modulated Lensless Imaging

In the past decade, mask-modulated lensless imaging systems have made great progress in Figure 1, and the most related progress is introduced according to the mask modulation types.

An amplitude-type mask encodes the incident light by partially obstructing or attenuating it. Pinhole imaging is the earliest form of optical imaging system and also the simplest form of amplitude-type mask modulation. However, due to the limited amount of light that passes through the pinhole, it results in a lower image quality and is less commonly used in modern imaging systems. With further research, Rego et al. [2] proposed a method to improve the quality of pinhole imaging. This method involves a joint denoising and deblurring image restoration process. The process combines low-pass filtering and denoising networks, which are jointly trained in a generative adversarial network (GAN), achieving the reconstruction of high-resolution static images and high-resolution videos.

To improve the signal-to-noise ratio of the imaging, researchers have developed encoded aperture imaging by using plates containing multiple transparent apertures instead of a single pinhole. This approach not only increases the transmissive area to enhance the light throughput and signal-to-noise ratio but also significantly reduces the physical size of the imaging system. Commonly used masks include band-limited coded masks [3], random aperture arrays [4], uniform redundant arrays (URAs) [5], and improved uniform redundant arrays (MURAs) [6].

However, MURAs can be impacted by the ill-posed nature of the inverse problem and the effects of diffraction. To address these issues, DeWeert and Farm [7] proposed Doubly-Toeplitz masks, in which point spread functions (PSFs) exhibit separable characteristics, making them more robust for wide-spectrum imaging. Asif et al. [8] designed the FlatCam camera, which places a binary mask with a 50% transmittance in close proximity to the image sensor, approximately 0.5 mm away. The PSF of FlatCam is essentially the projection of the mask itself. Adams et al. [9] introduced FlatScope, which positions the mask in front of the image sensor at a distance of 0.2 mm, achieving an imaging field of view of 6.52 mm² compared to just 0.41 mm² for traditional lens-based microscopes with the same sensor size. FlatScope also enables three-dimensional fluorescence microscopic imaging with micrometer-level resolution and a high frame rate. Shimano et al. [10] and Tajima et al. [11] proposed amplitude masks based on Fresnel zone apertures (FZAs), which are placed a few millimeters away from the image sensor. The aliasing effects are suppressed using stripe scanning, and digital refocusing is achieved through fast Fourier-transform (FFT). To mitigate the twin image problem in FZA-encoded imaging, Wu et al. [12] constructed a total variation regularization-based Fresnel-encoded aperture imaging reconstruction model. The sparse differences in gradient domains between twin images and the original image is considered. This approach reduces twin image noise and improves the imaging signal-to-noise ratio. Zhang et al. [13] introduced a binary mask-modulated lensless camera that uses multi-angle illuminations to recover the amplitude and phase information of the sample. Similar to the aperture constraints in Fourier ptychographic microscopy (FPM), the phase retrieval process satisfies the support constraints, and a 4.9 μm half-pitch resolution over a ~30 mm² field of view can be achieved.

Nakamura et al. [14] proposed an ultra-thin lensless camera that can capture super field-of-view images from both the front and back sides simultaneously. The camera consists of multiple coded image sensors, which are complementary metal–oxide–semiconductor (CMOS) sensors with random holes at some pixels. Zhang et al. [15] integrated a planar-coded aperture array with a sensor to create a lensless compound eye microsystem with an overall size of just 32 mm × 36 mm × 28.3 mm. The system can achieve high resolution, a wide field of view, and real-time sensing with a high frame rate. Hua et al. [16] proposed SweepCam, which can achieve depth perception of the scene by using a set of mask patterns displayed on a programmable mask. The computational focusing operator can help determine the depths of the scene points, and a fast multi-depth scene reconstruction algorithm is designed that can reduce the reconstruction time by two orders of magnitude. Zheng et al. [17] proposed a lensless three-dimensional imaging system that uses different patterns displayed on a programmable mask to capture the depth and intensity information of the scene. Moreover, a refinement network is devised as a postprocessing step to identify and eliminate artifacts in the reconstruction.

A phase-type mask modulates the phase of light using the principles of wave optics, with minimal obstruction or attenuation of the light, effectively addressing the issues of low light flux and low image signal-to-noise ratios encountered in amplitude-type masks. Phase gratings have a periodic structure that can induce a phase modulation of either 0 or $\mathsf{\pi }$ in the incident light. Stork et al. [18] introduced asymmetric phase-rotating gratings, integrating phase gratings with CMOS photodetector arrays to create ultracompact lensless PicoCam cameras. Antipa et al. [19] proposed DiffuserCam, which is based on imaging through a diffuser, and each point in the field of view corresponds to a unique pseudo-random pattern of defocused projections on the sensor, enabling single-frame three-dimensional imaging. Subsequently, Monakhova et al. [20] developed a hyperspectral camera by combining a diffuser with a filter array. The filter array encodes spectral information onto the sensor, and the diffuser maps each point in the real scene to multiple spectral filters for multiplexing. Boominathan et al. [21] designed a phase mask-based thin, lensless camera that can achieve high-resolution two-dimensional imaging, computational refocusing, and three-dimensional imaging. A contour-based high-performance PSF is proposed that utilizes signal processing concepts to achieve the maximum information transfer for finite bit-depth sensors. Moreover, it demonstrates how to reconstruct high-quality images and three-dimensional information from a single two-dimensional encoded measurement using fast linear methods and iterative nonlinear methods. Cai et al. [22] proposed a lensless light-field imaging method that uses a diffuser as an encoder, which maps the four-dimensional light-field information to a two-dimensional image. It establishes a diffuser-encoded light-field transmission model, which describes the propagation process of light through the diffuser and designs a calibration strategy to flexibly determine the transmission matrix. Through the transmission matrix, the rays can be computationally decoupled from the detected image, and the spatial and angular resolutions can be adjusted, breaking the resolution limit of the sensor. Baek et al. [23] proposed a lensless polarization camera that can achieve full-Stokes polarization imaging in a single shot, and the full-Stokes polarization imaging can provide additional contrast based on the birefringence and surface characteristics of objects. Moreover, a diffuser-encoded light-field transmission model is established, and an image reconstruction and calibration process is designed. Sitong et al. [24] proposed an “end-to-end” optoelectronics artificial intelligent framework for phase-modulated lensless cameras. Both intensity reconstruction and quantitative phase imaging for speckles and lens imaging are simulated with high accuracy. For speckle reconstruction using phase information, Pearson’s inner-product correlation coefficient (PCC) and peak signal-to-noise ratio (PSNR) can reach 0.929 and 19.313 dB. For speckle reconstruction using intensity information, the PCC and PSNR can reach 0.955 and 19.779 dB. For lens imaging with phase information, the PCC and PSNR can reach above 0.989 and 29.930 dB. For lens imaging with intensity information, the PCC and PSNR can reach above 0.990 and 30.287 dB.

The DOE, as a commonly used phase-type mask, can be designed with different PSFs, offering high degrees of freedom and ease of fabrication. However, they often suffer from chromatic aberrations, which can be mitigated through device optimization and reconstruction algorithm improvements in computational imaging. For instance, in device optimization, Peng et al. [25] proposed an achromatic diffractive lens. By balancing the focusing contributions of different wavelengths at the focal plane, they obtained a spectral invariant blurred kernel, achieving high fidelity and colored diffractive imaging across the entire spectrum. Zhao et al. [26] encoded the microstructure height of diffractive optical elements and optimized it using a particle swarm optimization (PSO) algorithm to achieve achromatic imaging in the visible light spectrum. Boominathan et al. [20] introduced a mask design with a contour line PSF that can produce high-contrast PSFs at different depths, which enables two-dimensional imaging, refocusing, and three-dimensional imaging. In joint optimization approaches, Dun et al. [27] simplified two-dimensional DOEs into one-dimensional rotationally symmetric structures to reduce the computational complexity. An end-to-end joint optimization method was employed. Baek et al. [28] developed differentiable simulators and neural network reconstruction methods, achieving single-frame high-spectral depth imaging through automatic differentiation. Additionally, stacking multiple diffractive elements is a common approach. Heide et al. [29] proposed stacking multiple DOEs and changing the alignment between them to achieve wide-spectrum imaging. Banerji et al. [30] devised achromatic planar multi-level diffractive lenses (MDLs) for infrared waves.

Figure 1. The development of the mask-modulated lensless camera in the last 10 years. In 2013, Stork et al. proposed an asymmetric phase-rotating gratings for the phase-type mask modulated camera—lensless PicoCam [18]. In 2015, DeWeert and Farm proposed separate Doubly-Toeplitz masks for amplitude-mask-modulated lensless imaging [7]. In 2016, Asif et al. devised the amplitude-type lensless camera—FlatCam [8]. In 2017, Antipa et al. developed DiffuserCam using a diffuser as the phase-type mask [19]. In 2017 and 2018, Tajima and Shimano et al. designed Fresnel zone apertures (FAZs) that included amplitude-type mask-modulated lensless imaging [10,11]. In 2018, Zhang et al. proposed a binary mask-modulated lensless imaging system that is inspired by Fourier ptychographic microscopy (FPM) [13]. In 2019, Nakamura et al. proposed an ultra-thin amplitude-type mask-modulated lensless camera [14]. In the next two years, the DiffuserCam [20], PhlatCam [21], and diffractive optical element (DOE) based diffractive camera [28] were designed which further enriched phase-modulated lensless imaging. Moreover, the SweepCam [16], FAZ [12], and lensless three-dimensional imaging system [17] were designed by using amplitude-modulated masks. In 2022, the compound eye microsystem [15] and lensless polarization camera [23] were devised for amplitude and phase-modulated lensless imaging systems, accordingly. In 2024, Chen et al. developed a learnable phase-modulated lensless camera based on the concepts of deep optics and diffractive optical neural networks [24].

Figure 1. The development of the mask-modulated lensless camera in the last 10 years. In 2013, Stork et al. proposed an asymmetric phase-rotating gratings for the phase-type mask modulated camera—lensless PicoCam [18]. In 2015, DeWeert and Farm proposed separate Doubly-Toeplitz masks for amplitude-mask-modulated lensless imaging [7]. In 2016, Asif et al. devised the amplitude-type lensless camera—FlatCam [8]. In 2017, Antipa et al. developed DiffuserCam using a diffuser as the phase-type mask [19]. In 2017 and 2018, Tajima and Shimano et al. designed Fresnel zone apertures (FAZs) that included amplitude-type mask-modulated lensless imaging [10,11]. In 2018, Zhang et al. proposed a binary mask-modulated lensless imaging system that is inspired by Fourier ptychographic microscopy (FPM) [13]. In 2019, Nakamura et al. proposed an ultra-thin amplitude-type mask-modulated lensless camera [14]. In the next two years, the DiffuserCam [20], PhlatCam [21], and diffractive optical element (DOE) based diffractive camera [28] were designed which further enriched phase-modulated lensless imaging. Moreover, the SweepCam [16], FAZ [12], and lensless three-dimensional imaging system [17] were designed by using amplitude-modulated masks. In 2022, the compound eye microsystem [15] and lensless polarization camera [23] were devised for amplitude and phase-modulated lensless imaging systems, accordingly. In 2024, Chen et al. developed a learnable phase-modulated lensless camera based on the concepts of deep optics and diffractive optical neural networks [24].

3. Image Reconstructions

The relationship between the sensor’s measurement values, denoted as “y”, and the target intensity, denoted as “x”, within the imaging model can be expressed as follows:

$$y=\mathrm{H}x+n$$

(1)

In this context, the matrix “H” depends on the specific design of the imaging system and various device parameters. These parameters include the pixel response function of the sensor, the choice in modulation devices, the distance between the sensor and the modulation device, the distance between the modulation device and the target, etc. The variable “n” represents the noise in the imaging system. Image reconstruction is essentially an inverse process, where the goal is to infer the target intensity from the measurement values.

For a given imaging system, assuming that the matrix “H” in the imaging model is accurately calculated and the sensor measurements “y” are obtained by capturing an unknown target, the objective of reconstruction algorithms is to estimate the unknown image “x” when provided with the sensor measurements “y” and the matrix “H”. However, in practical applications, model inaccuracies and system noise make solving the inverse problem challenging. Researchers have proposed various solution methods, with commonly used reconstruction algorithms falling into three categories: forward-solving algorithms, model-based optimization iterative algorithms, and deep learning algorithms. Here, we focus on introducing the deep learning algorithms.

3.1. Deep Learning Algorithms

Deep learning, as a data-driven method, requires datasets that fit real-world scenarios for learning. Currently, datasets can be collected through experiments or generated through simulations. Deep learning methods are generally divided into supervised learning, semi-supervised or partially supervised learning, unsupervised learning, and reinforcement learning, which is often discussed under partially supervised and unsupervised learning. Deep learning is mainly applied to solve image mapping problems and achieve image reconstruction. Among them, the U-Net convolutional neural network is widely used due to its simplicity and suitability for training with small datasets [31]. Transformer networks were first introduced in the paper “Attention Is All You Need” in 2017. Transformer networks are a type of neural network that can process sequential data. The Transformer model mitigates the shortcomings of convolutional neural networks (CNNs) (i.e., limited receptive field and inadaptability to input content). Instead, they use a mechanism called attention, which allows them to focus on the most relevant parts of the input and output sequences. With a comparative performance, it has been dramatically implemented in computer vision [32,33,34,35,36,37,38].

Typical lensless imaging systems include coded mask imaging systems and Fourier ptychographic imaging systems [39]. The combination of deep learning techniques with optical imaging methods can effectively improve the imaging speed and quality, and it has been successfully applied in various fields, such as digital holographic reconstruction [40,41,42,43], three-dimensional particle field imaging [44], phase recovery [45,46,47], and imaging through scattering media [48,49]. In recent years, deep learning techniques have also gradually been applied to lensless imaging.

In lensless imaging, Wu et al. [50] proposed a lensless three-dimensional imaging system that uses different patterns displayed on a programmable mask to capture the depth and intensity information of the scene. It uses a refinement network as a postprocessing step to identify and eliminate artifacts in the reconstruction in Figure 2a. Khan et al. [51] used trainable inversion layers to map measurements to an intermediate image space and then employed U-Net to improve the reconstruction, generating more realistic images. U-Net networks are also used for image depth estimations in Figure 2b, such as Wu et al. [52] using a U-Net network to obtain depth images from encoded images in Figure 2f. Baek et al. [28] proposed a differentiable simulator and a neural network-based reconstruction method, which were jointly optimized by automatic differentiation, thus achieving end-to-end learning of the mask and the image. Moreover, the paper built an experimental prototype and provided the first HS-D dataset, which was used to evaluate the performance of different deep learning algorithms on the lensless imaging system in Figure 2c. Zheng et al. [17] introduced a lensless imaging strategy that can image multiple stacks in the axial direction. The paper proposed a refinement network as a postprocessing step to identify and eliminate artifacts and obtain all-in-focus images in the reconstruction. The experimental results demonstrated that dense 3D scenes with a small number of sensor measurements can be recovered using a programmable lensless camera in Figure 2d. Rego et al. [2] introduced an imaging restoration pipeline for pinhole photography, which can reconstruct high-quality images from extremely low-resolution pinhole images. This pipeline uses deep learning and optimization techniques and consists of three steps: pinhole image calibration, super-resolution reconstruction, and color enhancement in Figure 2e. Furthermore, additional network modules can be added to existing networks to achieve specific functions. Monakhova et al. [53] proposed a network architecture called Le-ADMM in Figure 2g. The network unrolls the ADMM iterations into layers and learns the optimal ADMM parameters and the image reconstruction at the same time. Moreover, a U-net-based denoiser network can be added after this network to further improve the image quality.

3.2. Three-Dimensional Lensless Imaging Algorithms

Three-dimensional lensless imaging involves image compression algorithms, and compressive sensing (CS) is an effective technique for reconstructing a signal that has the property of transform sparsity by solving an underdetermined linear system using a small amount of sampled data [54].

To achieve the signal recovery from the observed measurements, the CS reconstruction problem can be formulated as a penalized model:

$$\widehat{x}=\arg \underset{x}{\min }\frac{1}{2}\parallel \mathrm{H}x-y{\parallel }_2^2+\mathcal{P}\left(x\right)$$

(2)

where $\mathcal{P}\left(\mathrm{x}\right)$ is a penalty term enforcing the transform sparsity of the signal. The purpose is to find the argument $\widehat{x}$ that minimizes the loss function. The least squares function is the most straightforward method to choose. As x in the least squares function is nonzero and not sparse, there are classical ways to modify the regression to achieve better properties, such as ridge regression or Tikhonov regularization. Moreover, the least absolute shrinkage and selection operator (LASSO) is proposed for even sparse signals in which one norm promotes sparsity [55].

For the optimization of the penalized model, a large variety of iterative algorithms have been developed, such as the fast iterative shrinkage-thresholding algorithm (FISTA) [56], Chambolle-Pock algorithm [57], the alternating direction method of multipliers (ADMM) [58], etc.

In three-dimensional lensless imaging, the complexity of both the calibration and computation can be reduced using a convolution model:

$$b\left(x^{\prime },y^{\prime }\right)={\displaystyle \sum }_{\left(x,y,z\right)}v\left(x,y,z\right)h\left(x^{\prime },y^{\prime };x,y,z\right)$$

(3)

where the object v is a set of point sources located at x, y, z on a non-Cartesian 3D grid. The caustic pattern of the pixels (x’,y’) on the sensor is the PSF, which can be expressed as $\mathrm{h}\left(x^{\prime },y^{\prime };x,y,z\right)$, and $\mathrm{b}\left(x^{\prime },y^{\prime }\right)$ is the sum of all the 2D sensor measurements.

Equation (3) can be further written as a L1-regularized least squares problem:

$$\begin{gathered} \widehat{v}=\underset{\omega \ge 0,u,v}{\mathrm{argmin}}\frac{1}{2}\parallel b-Dv{\parallel }_2^2+\tau \parallel u{\parallel }_1 \\ \hspace{1em}\hspace{1em}\hspace{1em}\mathrm{s}.\mathrm{t}. v=Mv,u=\mathsf{\Psi }u,\omega =v \end{gathered}$$

(4)

where $v$, $u$, and $\omega$ are auxiliary variables, D is a diagonal component, M and $\mathsf{\Psi }$ represent 3D convolutions, and $\tau$ is a tuning parameter that adjusts the degree of sparsity.

In general, the ADMM with variable splitting is employed for solving the 3D inverse problem of Equation (4), and the results at iteration k are

$$\begin{array}{c}{u}^{k+1}\leftarrow {\mathcal{T}}_{\frac{\tau }{{\mu }_2}}\left(\mathsf{\Psi }{v}^k+{\eta }^k/{\mu }_2\right)\\ v^{k+1}\leftarrow {\left(D^{\mathrm{T}}D+{\mu }_{1}I\right)}^{-1}\left({\xi }^k+{\mu }_{1}M{v}^k+D^{\mathrm{T}}b\right)\\ w^{k+1}\leftarrow \max \left({\rho }^k/{\mu }_3+v^k,0\right)\\ v^{k+1}\leftarrow {\left({\mu }_1{M}^{\mathrm{T}}M+{\mu }_2{\mathsf{\Psi }}^{\mathrm{T}}\mathsf{\Psi }+{\mu }_{3}I\right)}^{-1}{r}^k\\ {\xi }^{k+1}\leftarrow {\xi }^k+{\mu }_1\left(M{v}^{k+1}-v^{k+1}\right)\\ {\eta }^{k+1}\leftarrow {\eta }^k+{\mu }_2\left(\mathsf{\Psi }{v}^{k+1}-u^{k+1}\right)\\ {\rho }^{k+1}\leftarrow {\rho }^k+{\mu }_3\left(v^{k+1}-w^{k+1}\right)\end{array}$$

where

$$r^k=\left({\mu }_3{w}^{k+1}-{\rho }^k\right)+{\Psi }^{\mathbf{T}}\left({\mu }_2{u}^{k+1}-{\eta }^k\right)+M^{\mathrm{T}}\left({\mu }_1{v}^{k+1}-{\xi }^k\right)$$

(5)

Note that ${\mathcal{T}}_{\mathsf{\nu }}$ is a vectorial soft-thresholding operator with a threshold value of $\mathsf{\nu }$ [59], and ξ, η, and ρ are the Lagrange multipliers associated with v, u, and w, respectively. The scalars are penalty parameters that we compute automatically using the tuning strategy in [58].

Moreover, the minimization problem can also be described as the sum of the depths of interest with the total variation (TV) and L1 regularization terms as

$$\widehat{i}=\underset{i\ge 0}{\mathrm{argmin}}\parallel b-{\displaystyle \sum }_{d=1}^{\mathrm{D}}{P}_d\ast i_d{\parallel }_F^2+{\gamma }_1{\displaystyle \sum }_{d=1}^{\mathrm{D}}\parallel \Psi \left(i_d\right){\parallel }_1+{\gamma }_2\parallel i{\parallel }_1$$

(6)

where the number of depths D is calculated using the distance from the imaging device d and the 2D spatial gradient $\Psi$. As the TV and L1 regularization terms are applied to restrict under the sparsity assumption,${\gamma }_1$ and ${\gamma }_2$ are the tuning parameters that adjust the degree of sparsity. The 3D minimization problem can be further resolved by ADMM [60].

4. Related Techniques and Future Directions

The development of simple optical imaging relies on rapid advancements in various fields, such as optical field modulation devices, optical design software, image sensors, reconstruction algorithms, and artificial intelligence. However, it also presents several issues and challenges that need to be addressed.

As lenses have been used for imaging for centuries and they can create high-quality images, lensless imaging often produces lower-quality images than lens-based imaging. Lensless imaging systems need computational reconstruction algorithms to recover the unknown images of a scene. Thus, more processing power, lower energy consumption, and longer battery life are acquired. However, the light collection ability of a lensless camera is limited by the sensor size.

4.1. Deep Optics

The concept of computational imaging is to codesign the optics, sensing, and algorithms, and seminal works like wavefront coding, high dynamic range (HDR) imaging, super-resolutions, extend depths of field, light fields, and compressing imaging have been proposed in Figure 3. As designed in a compartmentalized fashion, deep optics is an end-to-end design that can be implemented to automate this process. In other words, deep optics jointly optimize optics and imaging processing end-to-end in simulations, and during the inference time, there is a minor adjustment for the imaging processing based on the fabrication of the optics. Furthermore, deep optics can be interpreted as an optical encoder and electronic decoder system in which the sensor is the bottleneck or a hybrid optical–electronic neural work in which the signals can be executed in photons or electrons, and low-level imaging processing in which physical parameters are included and high-level imaging processing can be optimized jointly. The applications of deep optics include low-light classification [61,62], monocular depth estimation [63,64], neural sensors [65,66], neural holography [67,68], and hybrid optical–electronic computing [69].

4.2. Diffractive and Optoelectronic Neural Networks

As shown in Figure 4, diffractive deep neural networks (D²NNs) have emerged as a free-space optical platform that leverages supervised deep learning algorithms to design diffractive surfaces for visual processing and all-optical computational tasks [70,71]. These diffractive optical networks, after their fabrication, form physical all-optical processors of visual information capable of executing various computer vision tasks spanning image classification [70,72,73,74,75], quantitative phase imaging (QPI) [76,77], linear transformations [78,79,80,81], image encryption [82,83], and imaging through diffusive media [84,85], among many others [86,87,88,89,90,91,92,93].

Each point on the diffractive layer represents an artificial neuron in which the phase or amplitude can be independently modulated, and the field distribution is

$$t_i\left(x_i,y_i,z_i\right)=a_i\left(x_i,y_i,z_i\right)\exp \left(j{\phi }_i\left(x_i,y_i,z_i\right)\right)$$

(7)

where $i$ denotes the neuron number, and the amplitude $a_i$ is assumed to be a constant. The phase term $\phi$ can be considered as a multiplicative “bias” term, which is a learnable parameter, and the Sigmoid function is used to limit the phase value of each neuron to 0~2 π.

Using the angular spectrum method, the transfer function can be expressed as

$$H\left(f_x,f_y,z\right)=\exp \left[j2\mathsf{\pi }z/\lambda \sqrt{1-{\left(\mathsf{\lambda }{f}_x\right)}^2-{\left(\mathsf{\lambda }{f}_y\right)}^2}\right]$$

(8)

where $z$ represents the axial distance between two positions, $\mathsf{\lambda }$ is the illumination wavelength,$f_x$ and $f_y$ are the frequencies in the x and y directions, and $j=\sqrt{-1}$.

The intensity distribution of the $i$th learnable neuron on the diffractive layer can be expressed as

$$I_i\left(x_i,y_i,z_i\right)={{\displaystyle \sum }}_k{\mathcal{F}}^{-1}\left\{\mathcal{F}\left[I_k^d\left(x,y,0\right)\right]·H_k^d\left(f_x,f_y,z_i\right)\right\}·t_i\left(x_i,y_i,z_i\right)$$

(9)

where $I_k^d\left(x,y,0\right)$ is the intensity distribution of the kth pixel on the diffusing layer. The field intensity right in front of the diffractive layer can be obtained by multiplying the transfer function $H_k^d$ with $I_k^d$ in the frequency domain and then making the summation over k in the spatial domain.

For optoelectronic neural networks, Shi et al. [94] introduced a lensless optoelectronic hybrid neural network architecture that can be used for computer vision tasks, such as handwritten digit classification and privacy-preserving face recognition. It uses a passive mask to perform convolution operations in the imaging optical path, thus extracting features in the optical domain and solving the challenge of processing incoherent and broadband light signals in natural scenes. This architecture also optimizes the mask design and the backend network to reduce the computation and energy consumption of the whole process. Wang et al. [95] introduced a lensless image sensing method that uses multilayer nonlinear optical neural networks (ONNs) as image preprocessors, which compress high-dimensional image information into a low-dimensional latent space for subsequent computational analyses, such as machine vision benchmarking, flow cytometry image classification, and object recognition in real scenes. It also shows that this ONN preprocessor can achieve high-precision and high-speed image sensing with fewer pixels and fewer photons under the same hardware conditions, which provides new ideas for developing compact and efficient lensless image sensing platforms.

4.3. Single-Pixel Imaging

Single-pixel imaging is a technique that generates images by illuminating a scene with a sequence of spatially structured patterns and recording the corresponding intensity on a detector that has no spatial resolution. Single-pixel camera architectures can be used for polarimetry [96], imaging through scattering media [97], phase imaging [98], and more. By using LCD, the lensless image can be further computed. Huang et al. [99] recently showed a lensless imaging system that used a LCD to control the light transmission at multiple apertures independently. This kind of single-pixel imaging system can avoid problems such as aberrations and produce images with an improved depth-of-focus by using point spread function engineering.

5. Applications

In recent years, with the rapid development of mask-modulated lensless imaging technology in the fields of three-dimensional and hyperspectral imaging and super-resolution and large FOV imaging, its peripheral applications in microscopy, endoscopy, data storage, and security have been accelerated in Figure 5. In addition to the amplitude and phase, the wavelength information of the light field can also be modulated to achieve hyperspectral lensless imaging. The lensless imaging systems with mask modulation enable high-throughput imaging. As a result, high-dimensional information such as the depth and spectrum can be obtained with both a large field of view and super-resolution, which is particularly useful for microscopic and endoscopic imaging. Furthermore, mask-modulated lensless imaging overcomes the “one-to-one projection” limitation and is suitable for compressed imaging. With the advantages of small size, light weight, and low cost, it can expand the application possibilities in data storage and security. The following sections introduce the scenario-based applications of the mask-modulated lensless imaging system.

5.1. Three-Dimensional and Hyperspectral Imaging

Depth imaging enables the acquisition of three-dimensional spatial information about a scene with wide applications, such as autonomous driving, robot obstacle avoidance, target recognition, and virtual reality. Simple optical depth imaging relies on monocular depth estimation methods, which restore depth information based on the varying degrees of object blur at different depths. Carvalho et al. [100] addressed depth estimation and 3D reconstruction from defocused images, i.e., real defocused indoor and outdoor images, presenting a complete system for single-image depth prediction through the defocus depth and neural networks.

Phase masks, by introducing depth-related point spread functions (PSFs), can strengthen the correlation between the scene depth and image blur. Wu et al. [50] proposed a method for passive 3D imaging using a phase mask at the camera aperture. This technique relies on an end-to-end optimization framework for jointly learning the optimal phase mask and reconstruction algorithm. It can accurately estimate distances from captured data and demonstrates excellent performance in 3D imaging. Boominathan et al. [21] designed a phase mask with a contour line PSF based on wave optics and phase recovery methods, achieving sharp PSFs at different depths for close range 3D imaging. Baek et al. [28] combined high-spectral and depth imaging by using DOE elements with PSFs that vary in depth and spectrum, enabling high-spectral-depth imaging in the visible light spectrum. Addressing the impact of DOE parameterization on the image quality and depth estimation accuracy for all-focus images, Liu et al. [101] found that PSFs with concentrated energy perform well for all-focus imaging, while PSFs with depth-related shapes are more suitable for depth imaging. Antipa et al. [19] generated pseudo-random PSFs using a diffuser plate and achieved single-frame 3D imaging by pre-calibrating PSFs at different depths. Tian et al. [102] used a custom microlens array to construct a camera, optimizing the PSF for the depth estimation and reconstructing three-dimensional scenes.

Lensless spectral imaging can capture the spectral information of a scene without using any lenses or moving parts. It relies on computational methods to reconstruct the hyperspectral data cube from a single or multiple measurements under different mask patterns. The mask patterns can be designed to encode multiplexed spatio-spectral information, which can be decoded by solving an inverse problem. Spectral imaging in lensless imaging has advantages, such as compactness and snapshot capability. It can be applied in various fields, such as medical imaging, remote sensing, food quality analysis, and astronomy [19,103].

5.2. Super-Resolution and Large FOV Imaging

Due to the limitations imposed by the diffraction limit and the influence of complex imaging environments, the resolution of images captured by imaging systems is inherently limited. To enhance image resolution, two approaches have been pursued: image reconstruction methods and optical methods aimed at surpassing the diffraction limits inherent in conventional imaging systems. Super-resolution image reconstruction involves the recovery of high-resolution images from low-resolution counterparts and has garnered considerable attention [104]. Tsai and Huang [105] explored the use of multiple image frames to achieve super-resolution reconstruction. Over the years, super-resolution techniques have matured, enabling the effective reconstruction of both video and static images. Sitzmann et al. [106] employed stochastic gradient methods to perform end-to-end optimization of optical elements based on the output of neural networks. The DOEs can achieve snapshot super-resolution imaging.

In the field of lensless microscopy, several techniques are employed for super-resolution imaging, such as subpixel displacement-based methods, multi-angle illumination, axial scanning, and wavelength scanning. The FlatScope lensless microscope achieves micrometer-level resolution for three-dimensional fluorescence imaging [9]. Additionally, bio-inspired microlens arrays are an effective means of achieving high-resolution imaging. Venkataraman et al. [107] introduced PiCam, an ultra-thin, high-performance, single-chip camera array capable of capturing light fields and obtaining high-resolution images and depth maps through integrated disparity detection and super-resolution. Wu et al. [108] presented an integrated scanning light-field imaging sensor, capturing ultra-fine four-dimensional light-field distributions through vibrating encoded microlens arrays, enabling high-resolution imaging and three-dimensional photography. Hu et al. [109] introduced a miniature optoelectronic-integrated compound eye camera capable of wide-field spatial imaging, recognizing the positions of moving objects, and tracking sensitive trajectories. Zhang et al. [15] proposed a lensless compound eye microsystem, stitching together multiple subfields formed by a telephoto sub-eye, and imaging panoramic targets within a single multiplexed image sensor. Moreover, the lensless compound eye microsystem employs a microelectromechanical systems (MEMS) aperture array to achieve wide-field and high-resolution imaging.

5.3. Microscopy

Lensless imaging offers a new way of undertaking microscopy with a small size and a flexible design. Compared to conventional microscopes [110], lensless imaging has different trade-offs among three factors: the FoV, resolution, and light collection efficiency. Conventional microscopes need large and heavy lenses with a high NA to collect more light and achieve a higher resolution, but they also have a smaller FoV [111]. Lensless imaging does not need any lenses, so it can have a larger FoV by using a bigger sensor or sensor array without losing the resolution. Lensless microscopy has shown its ability to capture large FoV and high-resolution images in various situations [9,112,113,114,115,116]. Lensless imaging also has other benefits for microscopy, such as 3D reconstruction and lesser size, weight, and cost.

Adam et al. [60] proposed a new type of lensless microscope that can image tissue in vivo using a phase mask that creates high-contrast contours in the diffraction patterns. The lensless microscope can achieve large fields of view, computational refocusing, and low cost and small form factors. The article also showed experimental results on imaging calcium dynamics in a mouse cortex, freely moving hydra, and oral microvasculature that suggested the lensless microscope may have clinical applications, especially for imaging hard-to-reach areas of the body in Figure 6a. Guo et al. [117] proposed a new on-chip imaging strategy, called depth-multiplexed ptychographic microscopy (DPM), that can image multiple specimen stacks in the axial direction at high speed and can achieve high-throughput imaging of biospecimens on a chip, such as cell cultures, brain sections, and blood smears in Figure 6b. Kuo et al. [118] introduced a lensless fluorescence microscope that only requires placing a diffuser on a conventional image sensor to achieve wide-field fluorescence imaging for planar and three-dimensional samples. The diffuser acts as a super-thin alternative to a microscope objective, making the system compact and easy to assemble, and also has a practical working distance of over 1.5 mm. Moreover, the diffuser also encodes volumetric information, enabling the refocusing and three-dimensional imaging of sparse samples during postprocessing. To improve the performance under low-light conditions, it also proposes a random microlens diffuser, which consists of many small lenses randomly placed on the mask surface, producing noise-robust PSFs Figure 6c.

5.4. Endoscopy

Fiber-based endoscopes have been around since 2011 that move away optics between fibers and samples and are consequently termed lensless endoscopes [119,120,121,122]. Cellular-level imaging in live mice by lensless endoscopes has been demonstrated using single-photon fluorescence imaging [123,124]. Sun et al. [125] proposed a new approach to 3D endoscopy using ultra-thin fiber bundles called multicore fibers (MCFs). Unlike conventional fiber facet imaging methods, where the imaging resolution is limited by the core-to-core spacing, their approach enables 3D quantitative phase imaging (QPI) reconstruction with nanoscale axial sensitivity of the optical path length and a lateral resolution up to 1 μm in the ideal case via direct recovery of the incident complex light field. Kuschmierz et al. [126] proposed two lensless fiber endoscopy techniques based on DOEs and deep neural networks that can achieve ultra-thin 3D imaging. The first technique is called the DOE-grating method, which uses DOEs designed and fabricated for phase conjugation and focusing to realize lensless grating scanning imaging. The second technique is called the DOE-diffuser method, which uses a diffuser to encode the far field of a coherent fiber bundle (CFB) into a two-dimensional speckle pattern and then decodes the speckle pattern by using a neural network to recover the 3D object information.

5.5. Data Storage and Security

Lensless imaging can be used to store data in a holographic memory, which is a type of optical data storage that uses holograms to encode information. Holographic memory can store more data than conventional optical discs and can also enable faster data retrieval. Hao et al. [127] proposed a deep learning-based phase retrieval method for comparative holographic data storage (HDS) with a higher storage density, lower bit error rate (BER), and higher data transfer rate. Hao et al. [128] proposed a novel method for recovering both the amplitude and phase information of data pages in holographic data storage using a deep learning model and a lensless near-field diffraction setup.

Lensless imaging can be used for privacy by capturing heavily blurred images that are imperceptible for humans to recognize the subject but contain enough information for machines to infer information. Sui et al. [129] proposed an optical multiple-image hiding method for a lensless optical security system. It considers various optical parameters as security keys, including propagation distance, wavelength, focal length and topological charge of the structured phase mask, which increases the size of the key space and improves the security.

It should be noted that mask-modulated lensless imaging depends largely on the backend image reconstruction algorithms, which affect the imaging quality, the power consumption, and the real-time display of the image reconstruction. Moreover, the light transmission in a mask-modulated lensless imaging system is restricted by the modulated transfer function of the mask and size and angular transfer function of the sensor. Hence, the optimization of lensless imaging architectures can be a promising direction to further extend their applications.

6. Future Outlook

The ongoing progress in lensless optical imaging technology has driven the development of various industries, including consumer electronics, autonomous driving, machine vision, security surveillance, biomedical applications, and the metaverse.

In the realm of consumer electronics, smartphone camera modules, often composed of multiple lenses, have been a subject of concern due to their protruding designs, impacting aesthetics. Lensless imaging systems offer new solutions by providing low SWaP (Size, Weight, and Power) imaging, enabling cost-effective, high-quality images and video capturing.

In the biomedical field, biomedical imaging and medical devices are the primary directions for the development and application of simplified optics. Lensless imaging could also be used to create high-resolution and large FOV images of biological samples, which can help in the diagnosis and treatment of diseases.

In the field of security, demanding clear and comprehensive images or videos for target monitoring, tracking, and attribute analysis, lensless imaging could be used to encrypt and decrypt data, which can enhance the security and robustness of data transmissions. Moreover, all-optical or optoelectronic lensless imaging can be implemented to deal with the need for improved concealment and reduced power consumption, which are essential to achieve comprehensive, all-weather surveillance.

In the field of autonomous vehicles, lensless imaging could be used to create compact and robust sensors that can detect and recognize objects in real time. For example, lensless imaging can be implemented to monitor a car’s surroundings, enabling panoramic vision, emergency collision avoidance, automated parking, and adaptive cruise control.

To better develop mask-based lensless imaging technology and its related fields, the following aspects require additional investigation. First, computing capability is essential for extracting information, and it also significantly increases the energy consumption requirements of the imaging system. Therefore, how to combine optoelectronic intelligent algorithms to achieve efficient computation is an important area that needs improvement. Second, the continuously improved and enhanced nanofabrication technologies are expected to reduce the cost of large size and high-precision optical modulation components. Third, by processing data directly at the sensor end, improving the area, time, and energy efficiency, and performing intelligent control of the pixel units, redundant data movements between the sensor and the reconstruction unit can be reduced, thus reducing the power consumption and time delay.