Hogel-Free Holography

2.1 Holographic Near-Eye Displays

Holographic displays have demonstrated powerful features such as compact form factor, variable focus control, and aberration correction, all of which are essential for eye-wear displays. In order to reduce the form factor, researchers have used multi-functional Holographic Optical Elements (HOEs) [Maimone et al. 2017; Li et al. 2016; Jang et al. 2017, 2019] and waveguides [Yeom et al. 2015] to relay the projected imagery into the eye. Holographic displays typically suffer from a tiny eyebox and poor image quality. However, significant advances are already being made in recent years to solve these problems. Recent approaches such as increasing the eyebox size by using eyetracking [Jang et al. 2017, 2019] and employing specially designed HOEs [Xia et al. 2020; Jang et al. 2019] or scattering elements [Kuo et al. 2020; Park et al. 2019] show tremendous promise for achieving practical displays in the future. In addition, employing eyetracking [Lu et al. 2020] might further decrease computational burden [Deng et al. 2021] and facilitate reflectance, focus control, and aberration correction [Chakravarthula et al. 2021], overcoming challenges related to the limited space-bandwidth product of existing SLMs. While recent algorithmic advances have made it possible to generate high-quality 2D holograms [Chakravarthula et al. 2020; Peng et al. 2020], modest image quality of 3D holographic projections with visible artifacts [Shi et al. 2021] remains as a critical limitation of today’s holographic displays.

2.2 2D Holographic Phase Retrieval

Complex holograms for a given target image can be computed by propagating the light waves emanating from the scene to the SLM (hologram) plane, where they are interfered with the reference beam [Benton and Bove Jr 2008]. While this hologram field on the SLM is complex and has both amplitude and phase, existing SLMs can only modulate either amplitude or phase but not both, simultaneously. As phase-only SLMs are typically preferred for better efficiency, this necessitates the computation of phase-only holograms that are capable of producing the diffraction field that most closely mimics the target image. In order to do so, a variety of techniques have been proposed which can be broadly categorized into direct and iterative phase retrieval methods. Direct methods attempt to encode the complex wavefield at the SLM into a phase-only representation using amplitude-phase coding [Hsueh and Sawchuk 1978; Burckhardt 1970; Lee 1970] but typically result in reduced image quality. Iterative methods, on the other hand, improve the image quality by minimizing the reprojection error of the target intensity distribution using iterative optimization but are usually slow [Gerchberg 1972; Chakravarthula et al. 2019]. Remarkably, high-quality 2D holograms have been recently demonstrated on experimental prototypes using hardware-in-the-loop optimization [Chakravarthula et al. 2020; Peng et al. 2020].

2.3 3D Holographic Phase Retrieval

Algorithms for 3D computer-generated holography have been the focus of active research for several years. The most popular existing methods use RGB-D or multiplane images as shown in Figure 2 to represent 3D objects. These methods fail to accurately model wavefronts in the presence of occlusions in the scene. We overcome this challenge by representing the 3D scene as a light field which implicitly models occlusions and parallax, thereby also modeling the wavefronts emanating from occluded scene parts. In this section, we discuss existing 3D holographic phase retrieval methods as summarized in Figure 1 .

Fig. 1.

View Figure

Fig. 2.

View Figure

Point Cloud and Polygonal Mesh-Based Methods. Researchers have leveraged 3D scene representations such as point clouds or polygonal meshes for computing holograms [Waters 1966; Leseberg and Frère 1988], where diffraction is simulated for every scene point or polygon (see Figure 2(a)). Computing such holograms is generally computationally expensive and results in a complex field at the SLM plane. This complex field is then converted into a phase-only hologram using heuristic encodings such as double-phase amplitude encoding [Maimone et al. 2017; Hsueh and Sawchuk 1978]. Similarly, a look-up table of elemental fringes can also be precomputed [Lucente 1993; Shi et al. 2017]. Existing encoding schemes have been shown to result in reduced spatial resolution and image quality [Chakravarthula et al. 2019].

Multiplane Images and Focal Stack–Based Methods. Instead of computing the wave propagation for millions of points, a 3D object can be represented as a stack of intensity layers [Zhang et al. 2016; Zhao et al. 2015; Choi et al. 2021; Eybposh et al. 2020] (see Figure 2(b)). Wave propagation methods such as the inverse Fresnel transform or angular spectrum propagation are typically used for propagating the waves from several layers of the 3D scene toward the SLM plane, where they are interfered to produce a complex hologram. Although this approach can be implemented efficiently, it cannot support continuous focus cues and accurate occlusion due to discrete plane sampling [Kuo et al. 2020; Peng et al. 2020; Zhang et al. 2017; Makowski et al. 2007]. A further approximation to the layer-based methods is to determine the focal depth of the user (i.e., distance of the object to which the user fixates) via an eyetracker and adjusting the focal plane of the 2D holographic projection to match the user’s focal distance [Maimone et al. 2017; Chakravarthula et al. 2020]. While emulating a 3D scene by adaptively shifting a 2D holographic projection in space is computationally more efficient, operating in a varifocal mode under-utilizes the capabilities of a holographic display. Moreover, achieving natural focus cues and physically accurate occlusion effects still remains a challenge.

Light Field–Based Methods. To support occlusion and depth-dependent effects, a light field can be encoded into a hologram partitioned spatially into elementary hologram patches, called “hogels” [Zhang et al. 2015], similar to elementary images in a light field (see Figure 2(c)). These hogels produce local ray distributions that reconstruct multiple (light field) views [Lucente and Galyean 1995; Yamaguchi et al. 1993; Smithwick et al. 2010]. Such holograms which encode a light field are dubbed “holographic stereograms.” Conventional stereograms, where hogels are out of phase with each other, suffer from a lack of focus cues and limited depth of field [Lucente and Galyean 1995]. To keep the hogels of a holographic stereogram in phase across the hologram, researchers have introduced an additional phase factor to calculate what is called a phase-added stereogram (PAS) [Yamaguchi et al. 1993]. However, akin to a microlens array-based light field display [Lanman and Luebke 2013], stereograms suffer from the fundamental spatio-angular resolution tradeoff: a larger hogel size leads to a decreased spatial resolution. This fundamental limitation does not allow for holographic stereogram projections of high spatial resolution. However, recent methods have attempted to overcome this tradeoff [Padmanaban et al. 2019; Blinder and Schelkens 2018; Park and Askari 2019] via Short-Time Fourier Transform (STFT) inversion. These methods do not match the image quality achieved for 2D holograms [Chakravarthula et al. 2020; Peng et al. 2020] and suffer from severe artifacts around object discontinuities due to sub-optimal inversion of the STFT. In this work, we lift these limitations and demonstrate an iterative optimization method for 3D holographic phase retrieval (see Figure 2(d)).

3.1 Input Scene Representation

We use an RGB-D light field as input to our method because this representation contains parallax and mutual occlusion information. As depicted in the bottom row of Figure 3, light field arrays consist of multiple views of the scene which allows us to capture missing scene content from partial occlusions for a single camera view. In contrast, commonly employed scene representations such as RGB-D images (Figure 3, middle row) and multiplane images (Figure 3, top row) insufficiently describe the scene due to depth discontinuities and missing information in the presence of occlusions. Specifically, for computing the 3D hologram, every point on the visible scene is assumed to be a spherical wave emitter whose diffraction pattern is independently superposed on the SLM. As a result, wavefronts in the presence of occluders are inaccurately modeled resulting in visible artifacts at the occlusion boundaries (see Figure 3). Existing methods relying on single-view RGB-D images or multiplane images as inputs ignore diffraction at depth discontinuities, causing severe ringing artifacts and noise. Note that focal stacks suffer from the same limitations.

Fig. 3.

View Figure

3.2 Consensus 3D Holograms from Light Fields

Our aim is to directly optimize the SLM phase pattern for a full propagating wavefront of a 3D scene as opposed to existing light field–based stereogram approaches which solve piecewise planar approximations of the wavefronts represented by hogels, as depicted in Figure 4. In other words, for a given light field \( L(x,y,\theta _x,\theta _y) \) parameterized by positions \( x,y \) and angles \( \theta _x,\theta _y \), we want to find a phase-only SLM pattern \( \Phi \) that minimizes the following objective: (1) \( \begin{equation} \begin{aligned}\Phi _{\rm\small OPT} = \hspace{4.2679pt} & \underset{\Phi }{\text{min}} \sum _{ \lbrace \theta _x, \theta _y\rbrace } \Big |\Big | \hspace{4.2679pt} \big | \mathcal {\sigma }_{(\theta _x,\theta _y)} \lbrace \mathcal {P} (e^{j\Phi }, z_{\rm\small WR}) \rbrace \big |^2 \hspace{4.2679pt} - L(\theta _x,\theta _y) \Big |\Big |, \\ \end{aligned} \end{equation} \) where \( U=e^{j\Phi } \) is the wavefield at the SLM, \( \mathcal {P}(U,z_{\rm\small WR}) \) is a wave propagation kernel, such as angular spectrum propagation, which propagates the input field U to the wave recorder (WR) plane at the distance \( z_{\rm\small WR} \) where the light field is measured, and \( \begin{equation*} \sigma _{(\theta _x,\theta _y)}: U(x,y) \longrightarrow L(x,y,\theta _x,\theta _y) \end{equation*} \) is the wavefront sampling operator parameterized by the light field angular views \( \theta _x,\theta _y \). The positions \( x,y \) are ignored in Equation (1) for brevity. The operator \( \sigma _{(\theta _x,\theta _y)} \) and its inverse \( \sigma ^{-1}_{(\theta _x,\theta _y)} \) are detailed in the following sections.

Fig. 4.

View Figure

For an \( M\times N \) light field input, performing the above optimization in the light field domain requires optimizing over MN summation terms. While such an objective may be minimized using consensus optimization [Boyd et al. 2004], we instead solve it in the wave domain in order to pose the objective as a single term. Specifically, the objective in the wave domain becomes (2) \( \begin{equation} \begin{aligned}\Phi _{\rm\small OPT} = & \underset{\Phi }{\text{min}} \Big |\Big | \mathcal {P} (e^{j\Phi }, z_{\rm\small WR}) - \sum \mathcal {\sigma }^{-1}_{(\theta _x,\theta _y)} \Big \lbrace \sqrt {L(\theta _x,\theta _y)} e^{j\Psi (\theta _x,\theta _y)} \Big \rbrace \Big |\Big |, \end{aligned} \end{equation} \) where \( \sigma _{(\theta _x,\theta _y)}^{-1} \) maps discrete light field angular views to their corresponding complex wavefront description. Note that light fields are intensity-only images and the associated phase \( \Psi (\theta _x,\theta _y) \) corresponding to each angular view of the light field is typically missing, which makes solving the above problem by inverting \( \sigma \) ill-posed.

Furthermore, besides being ill-posed, notice that Equation (2) is defined in the complex domain. As most commodity SLMs only support phase modulation, existing 3D holography methods have to resort to heuristic phase encoding schemes that turn the complex solution into a phase-only solution.

To overcome both challenges in solving Equation (2), we ameliorate the complex domain objective by relaxing the target phase constraints in Equation (2) into a continuous volume amplitude constraint and solve it using stochastic gradient descent to find a phase-only true 3D hologram. We describe this approach in detail in the following sections (along with operators \( \sigma _{(\theta _x,\theta _y)} \), \( \sigma ^{-1}_{(\theta _x,\theta _y)} \)). Specifically, Section 3.3 expresses \( \sigma \) in Equation (1) as a Windowed Fourier Transform (WFT) and Section 3.4 describes how we cure the ill-posed inversion of \( \sigma \) described in Equation (2). Finally, Section 3.5 describes how we relax the phase constraints in Equation (2) and how we pose the 3D phase retrieval as an optimizing amplitude across a continuous 3D volume.

3.3 Light Field Forward Model

We will describe how a source wavefield \( U_{\rm\small SRC}(u,v;0) \) is related to a given light field \( L(x,y,\theta _x,\theta _y) \) measured on a certain wave recorder plane at a distance \( z_{\rm\small WR} \): (3) \( \begin{equation} U_{\rm\small SRC}(u,v;0) \longrightarrow U_{\rm\small WR}(x,y;z_{\rm\small WR}) \longleftrightarrow L(x,y,\theta _x,\theta _y), \end{equation} \) where \( U_{\rm\small WR}(x,y;z_{\rm\small WR}) \) is the underlying complex wavefront on the wave recorder plane. We formalize the forward pass \( U_{\rm\small SRC}(u,v) \longrightarrow L(x,y,\theta _x,\theta _y) \), that is, how a source field is propagated from a scene to a wave recorder and captured as a light field. To this end, let us recall angular spectrum decompositions [Goodman 2005]: The angular spectrum propagation of a source field \( U_{\rm\small SRC}(u,v;0) \) by some distance z is described by (4) \( \begin{equation} U_{\rm\small WR}(x,y;z) = \mathcal {F}^{-1} (\mathcal {F} \lbrace U_{\rm\small SRC}(u,v;0) \rbrace \circ H(f_x, f_y;z)), \end{equation} \) where \( U_{\rm\small WR} \) is the field on a WR plane where the light field is measured and H is the angular spectrum propagation kernel given by (5) \( \begin{equation} H(f_x, f_y; z) = {\left\lbrace \begin{array}{ll}\exp { \Big [ j2\pi \frac{z}{\lambda } \sqrt {1-(\lambda f_x)^2 - (\lambda f_y)^2} \Big ]},&\hspace{-8.0pt} \sqrt {f_x^2 + f_y^2} \lt \frac{1}{\lambda } \\ 0,&\hspace{-8.0pt} \text{otherwise} \end{array}\right.} \end{equation} \) with z being the propagation distance of the modulated monochromatic wave of wavelength \( \lambda \) with spatial frequencies \( f_x, f_y \) in the x and y directions, respectively. The Fourier transform \( \mathcal {F}(.) \) in the above Equation (4) decomposes the source wave field into its component plane waves, that is, the angular spectrum.

The spatial frequencies \( (f_x,f_y) \) present in the angular spectrum relate to the direction cosine/light field camera view angles \( (\theta _x, \theta _y) \) [Goodman 2005] as (6) \( \begin{equation} f_{\lbrace x,y\rbrace } = \frac{\sin (\theta _{\lbrace x,y\rbrace })}{\lambda }. \end{equation} \) Therefore, to model wavefronts in the presence of partial occlusions via light fields, it is sufficient to model local plane waves with corresponding spatial frequencies as described by Equation (6). However, every spatial frequency component of the angular spectrum extends over the entire \( (x,y) \)-domain, but only sparse discrete views are available from the input light field. However, note that a light field is nothing but a coerce representation of a continuous wave field. As a result, the underlying holographic wavefield is capable of representing continuous novel views and at any focal plane, these views are aggregated to form an appropriately defocused image.

Wavefront View Sampling. To tackle the mismatch between discrete light field views and the dense frequency support, locally sampling the propagated wavefront \( U_{\rm\small WR} \) offers a solution. The spatially localized frequency distribution corresponding to light field angular views can be obtained by spatially limiting the Fourier spectrum to only a constrained set of frequencies, which we do by using a windowing function \( w(x,y) \) that is non-zero only for a limited support [Ziegler et al. 2007]. The resulting local frequency spectrum can be computed by a WFT as (7) \( \begin{equation} \begin{split} S&(x,y,f_x,f_y) = \sigma _{(\theta _x,\theta _y)} \lbrace U_{\rm\small WR}(x,y) \rbrace \\ &\times \iint \limits _{-\infty }^{\infty } U_{\rm\small WR}(x^{\prime },y^{\prime }) w(x^{\prime }-x,y^{\prime }-y) e^{-j2\pi (f_x x^{\prime } + f_y y^{\prime })} \,dx^{\prime } \,dy^{\prime }. \end{split} \end{equation} \) Multiplying the field on the wave recorder \( U_{\rm\small WR} \) with the window function w (e.g., Hamming window) suppresses the field outside the window, causing localization. The light field L measured on the wave recorder plane can then be described as the squared magnitude of the angular distributions of the local frequency spectrum S, that is, (8) \( \begin{equation} \begin{split} L(x,y,\theta _x,\theta _y) & = | S(x,y,f_x,f_y)|^2 \\ & = | \mathrm{WFT}\lbrace U_{\rm\small WR}(x,y) \rbrace (x,y,f_x,f_y)|^2. \end{split} \end{equation} \) In other words, we can formulate our RGB-D light field forward model from Equation (1) with \( \sigma \) being the WFT.

3.4 Ill-Posed Light Field Inversion

After having modeled the input light field in terms of the underlying wavefield, we now describe the inversion of the forward model \( \sigma ^{-1}:L(x,y,\theta _x,\theta _y) \longrightarrow U_{\rm\small WR}(x,y) \) that was shown in Equation (2). It can be seen from Equation (7) that the windowed Fourier transform of a wavefield results in a complex spectrum containing both amplitude and phase. The wavefield can be recovered by naively inverting the discretely sampled spectrum from Equation (7) in the frequency domain as (9) \( \begin{equation} \begin{split} U_{\rm\small WR}(x,y) &= \iint \limits _{-\infty }^{\infty } w(x-x^{\prime },y-y^{\prime }) \iint \limits _{-\infty }^{\infty } L(x^{\prime },y^{\prime },\theta _x,\theta _y) \\ &\quad \times e^{ j (2\pi (f_x x^{\prime } + f_y y^{\prime }) + \Phi (x,y,f_x,f_y)) } df_x df_y dx^{\prime } dy^{\prime } \\ &= \mathrm{WFT}^{-1} \Big \lbrace \sqrt {L(x,y,\theta _x,\theta _y)} e^{j \Psi (x,y,f_x,f_y)} \Big \rbrace , \\ \end{split} \end{equation} \) where \( \Psi (x,y,f_x,f_y) \) is the associated phase and w is the window function as described in Equation (7). Note that this relation can also be obtained via Wigner distribution function [Zhang and Levoy 2009] and solved via stereogram approaches [Padmanaban et al. 2019].

This reveals a problem, as shown in Equation (8); a captured or rendered light field contains only amplitude information and does not describe the phase \( \Psi \). This makes the inversion from Equation (9) ill-posed.

We alleviate this by defining the missing phase using the depth maps associated with the RGB-D light field array. Specifically, we evaluate the path delays along each individual angular light field direction. These path delays are obtained via the depth fields \( D(x,y,\theta _x,\theta _y) \) associated with the light field intensities described by \( L(x,y,\theta _x,\theta _y) \) [Yamaguchi et al. 1993]. We then set the missing phase information \( \Psi \) to (10) \( \begin{equation} \frac{2\pi }{\lambda } D(x,y,\theta _x,\theta _y). \end{equation} \) We plug the target phase from Equation (10) into Equation (9) so that we can perform the light field inversion described in Equation (2). And in doing so, we obtain the target wavefield \( U_{\rm\small WR} \). This resolves the ill-posed inversion.

3.5 Phase-Relaxed 3D Holography

With the forward and inverse of \( \sigma \) formulated above, the complex domain objective in Equation (2) is fully defined. However, the optimization objective in Equation (2) penalizes both amplitude and phase and formulating an appropriate penalty on phase is still a challenge. Recall that the phase of a wavefront at any given distance directly relates to the amplitude of that wavefront over a continuous 3D volume. For example, Figure 5 shows the amplitude of the propagated wave with different initial phases (shown in the inset). The wavefronts with a uniform initial phase on the aperture plane (first row) evolve differently from that of a linear (second row) or random phase (last two rows). This dependence of the wave propagation on the initial phase allows us to convert a phase objective into an amplitude objective. Specifically, we pose the objective as an amplitude optimization over the continuous 3D volume as (11) \( \begin{equation} \begin{aligned}\Phi _{\rm\small OPT} = \hspace{4.2679pt} & \underset{\Phi }{\text{min}} \int _{V} \textstyle \mathcal {L} \hspace{1.13809pt} \Big (\hspace{2.84526pt} \big | \mathcal {P}(e^{j\Phi }, \hspace{1.13809pt} z_{\rm\small WR} + z) \big |, \hspace{1.13809pt} \big | \mathcal {P}(U_{\rm\small WR}, \hspace{1.13809pt} z) \big | \Big) dz, \end{aligned} \end{equation} \) where \( \mathcal {L} \) is a custom penalty function, described further below, which we evaluate on the amplitudes over the continuous 3D volume V, \( \Phi \) is the SLM phase, and \( U_{\rm\small WR} \) is the complex wavefront measured at the wavefront recorder plane where the light field is measured as described in Section 3.4.

Fig. 5.

View Figure

With the objective from Equation (11), we can optimize 3D holograms over a continuous volume, for the first time, to the best of our knowledge. Note that this approach does not explicitly calculate hogels corresponding to the input light field in the final optimized hologram, making it hogel-free. Also observe that our method can support any depth range.

Continuous Volume Optimization. Algorithm 1 reports our optimization method to solve Equation (11). See Figure 6 for an illustration of the algorithm. Our algorithm solves the objective via stochastic gradient descent. Specifically, an RGB-D light field is provided as input and the optimizer is initialized with a random phase (see line 3). Line 4 then applies the inverse of \( \sigma \) described in Section 3.4 to convert the light field into the underlying wavefront \( U_{\rm\small WR} \) on the wave recorder. Line 6 randomly samples depths from a uniform distribution for every iteration within the continuous volume. We propagate the target complex field \( U_{\rm\small WR} \) throughout a continuous depth volume using an angular spectrum wave propagation function \( \mathcal {P}(U_{\rm\small WR}, z) \). We then propagate the phase-only hologram field at the SLM \( U_{\rm\small SLM} = e^{j \Phi } \) (initialized with random phase) as \( \mathcal {P}(U_{\rm\small SLM}, z_{\rm\small WR} + z) \) toward where the target amplitude from the complex wavefield \( U_{\rm\small WR} \) is evaluated. As shown in line 7, the penalty is computed only on the amplitude of the wavefronts at these sampled depths. Then the error is backpropagated into the phase, as shown in line 8, where the SLM phase is updated until convergence.

Fig. 6.

View Figure

Penalty Function. The custom penalty function \( \mathcal {L} \) can be freely defined as a function of observed image amplitude or intensity to facilitate optimization of perceptually appealing 3D holograms. To that end, we use a weighted combination of \( \ell _2 \) penalty \( \mathcal {L}_{\ell _2} \), SSIM \( \mathcal {L}_{\rm\small SSIM} \), perceptual penalty based on VGG-19 \( \mathcal {L}_{\rm\small PERC} \) [Johnson et al. 2016], and Watson FFT \( \mathcal {L}_{\rm\small WFFT} \) [Czolbe et al. 2020]: (12) \( \begin{equation} \mathcal {L} = \lambda _{\ell _2}\mathcal {L}_{\ell _2} + \lambda _{\rm\small SSIM}\mathcal {L}_{\rm\small SSIM} + \lambda _{\rm\small PERC}\mathcal {L}_{\rm\small PERC} + \lambda _{\rm\small WFFT}\mathcal {L}_{\rm\small WFFT}. \end{equation} \) We use a least-square penalty \( \mathcal {L}_{\ell _2} \) for per-pixel accuracy in the reconstruction and \( \mathcal {L}_{\rm\small SSIM} \) as a handcrafted perceptual quality function. The perceptual penalty \( \mathcal {L}_{\rm\small PERC} \) compares the image features from activation layers in a pre-trained VGG-19 neural network, that is, (13) \( \begin{equation} \mathcal {L}_{\rm\small PERC} = \sum _{l} v_l \left\Vert \phi _l(x) - \phi _l(y) \right\Vert _1\!, \end{equation} \) where \( \phi _l \) is the output of the l-th layer of the pre-trained VGG-19 network and \( v_l \) are the corresponding penalty-balancing weights. Specifically, we use the outputs of ReLU activations just before the first two maxpool layers, i.e., relu1\( \_ \)2 and relu2\( \_ \)2. Adopting \( \mathcal {L}_{\rm\small PERC} \) therefore helps achieve finer details in the reconstructed image. However, note that the VGG-19 network [Simonyan and Zisserman 2015] is optimized for classification and detection tasks, and is robust to the perceptual influence of artifacts such as noise. As a result, using a VGG-based perceptual penalty does not always guarantee perceptually high-quality results. Therefore, we further improve the reconstruction quality by adopting the Watson FFT error function which is crafted specifically for human visual system, based on Watson’s visual perception model [Watson 1993]. The combination of the above losses helps steer the optimization toward holograms that are perceptually pleasing for the human visual system. The perceptual losses mitigate perceptually apparent reconstruction noise across the 3D volume.

4.1 Implementation

We test our hogel-free holography approach in simulation and implement the entire optimization framework within PyTorch running on a single Nvidia GeForce GTX 1080 GPU providing 8 GB of memory. PyTorch now provides complex gradients within its auto-differentiation modules making implementation of our optimization scheme with state-of-the-art first-order optimizers straightforward. We notice that different optimizers result in slightly different reconstruction quality. We optimize for \( 1{,}080 \times 1{,}920 \) phase-only holograms, same as our SLM resolution, using the Adam optimizer running for 800 iterations with a learning rate of 0.1. This optimization currently takes about 300 s. Our 3D scene dataset consists of light field and depth field images rendered at 1,080p in Unity as well as those from Padmanaban et al. [2019] to accommodate the SLM maximum diffraction angle. For computing the WFT we use a \( 9\times 9 \) Hamming window to match the number of light field views. The depth range of our 3D scenes extends throughout a 13 mm volume; however, note that this depth range is not fundamental to the method.

4.2 Experimental Prototype

To validate our simulations, we built an experimental benchtop prototype with a HOLOEYE LETO 1,080p phase-only LCoS SLM with a pixel pitch of \( 6.4 \,\mathrm{\mu }{\rm m} \) (Figure 7). We illuminated the SLM using a collimated beam from a single optical fiber that is coupled to three laser diodes emitting at nominal wavelengths of 630 nm, 520 nm, and 450 nm. The laser light was linearly polarized, matching the requirements of the SLM. The SLM was connected to a PC running on 64-bit Windows OS. However, the look-up-table adjustments to the SLM were made from a 32-bit Windows machine due to the age of our SLM. We use a telescopic optical relay system to image the holographic 3D projections with a Canon CMOS APS-C sensor. Additionally, we also placed an iris at an intermediate image plane to discard unwanted diffraction orders arising from the double phase encoding of OLAS [Padmanaban et al. 2019]. The RGB holographic images were captured in a color sequential manner with the camera settings unchanged but laser powers tuned to approximately white balance the illumination.

Fig. 7.

View Figure

5.1 Comparison with Existing Stereogram Approaches

We evaluate the proposed Hogel-Free Holography in simulation and we demonstrate a significant improvement over existing light field based state-of-the-art stereogram methods. Specifically, we compare against the holographic stereograms [Yatagai 1976], the phase-added stereograms [Yamaguchi et al. 1993], and the overlap add stereograms [Padmanaban et al. 2019]. We see in Figure 8 that our method closely matches the target 3D scene despite modulating only the phase of a simulated laser beam. Qualitative comparisons against baseline stereogram methods are shown in Figure 8 and we observe visible improvement at both near and far focus.

Fig. 8.

View Figure

For quantitative evaluation, we compute the PSNR, SSIM, and LPIPS [Zhang et al. 2018] metric scores on the depth planes within each 3D scene. The average metric scores computed at depths sampled across the continuous volume are reported in Table 1. We observe over 10 dB improvement in PSNR, significantly outperforming the existing methods. Also, note that the LPIPS perceptual similarity metric shows significant improvement compared to previous methods, validating perceptually high-quality optimization of 3D holograms using our method.

5.2 Comparison with Occlusion-Aware Tensor Holography

Here, we compare our approach with the tensor holography method [Shi et al. 2021]. Tensor holography is a real-time neural network approach that implements a ray visibility-based occlusion-aware point-based method. Figure 10 shows qualitative comparisons of our method with holograms produced using tensor holography. As discussed in Section 3, ray visibility–based methods do not model diffraction due to hard occlusion edges and depth discontinuities in a physically accurate manner. As a result of this inaccurate occlusion modeling, light leaks from the background into the foreground severely contaminating the image with visible ringing artifacts, as can be seen in Figure 10. For example, observe the edges of the blades of grass where ringing artifacts can be clearly seen.

The physically inaccurate model in tensor holography also causes incorrect defocus effects, as shown in Figure 10. Since tensor holography generates holograms based on a single RGB-D image, the single perspective depth information is not always sufficient for computing accurate 3D holograms. As an example, we compare the 3D holographic reconstructions of a 3D scene visible from a non-Lambertian mirror surface in Figure 9. Note that the depth map of the mirror is rendered as a planar surface, without providing any detail of the 3D scene shown within the mirror. As a result, reconstructions produced by tensor holography do not exhibit accurate defocus effects. Also, ringing due to light leakage from the background can be seen near the candles. The proposed hogel-free holography produces accurate defocus effects in this challenging scene as the defocus and occlusion cues are implicitly encoded from the light field.

Fig. 9.

View Figure

5.3 Robustness to Hardware Non-Idealities

Holographic displays unfortunately possess several deviations from the ideal light transport model [Chakravarthula et al. 2020; Peng et al. 2020]. This results in substantial disparity between simulated reconstructions and real holographic display output. We demonstrate that holograms generated by our HFH are more robust to these non-idealities than the previous state-of-the-art OLAS. Figure 11 shows an example where we simulate the effect of a non-linear SLM look-up table by introducing a gamma factor of 0.8 on the simulated phase. This phase non-linearity incurs a significant drop in quality for OLAS, both qualitatively and quantitatively, while our proposed HFH is much more tolerant of the deviation, validating the robustness of our method. Additional details can be found in the Supplementary Information.

6.1 Experimental Validation

Figures 12 and 13 showexperimentally acquired results from the prototype display described in Section 4. The results in Figure 12 demonstrate that the proposed approach mitigates most of the severe artifacts present in existing light field–based stereogram methods. Specifically, spatio-angular resolution tradeoff due to the use of hogels in computing holographic stereograms limited the achieved spatial resolution of previous methods. While the overlap-add stereograms attempt to overcome this tradeoff, the usage of double-phase amplitude coding (DPAC) leads to visible artifacts and a loss of spatial resolution. Furthermore, DPAC leads to a checkerboard pattern on the hologram phase which results in the unwanted effect of a noticeable portion of light escaping the image window into higher diffraction orders. This creates multiple higher-order ghost images which leads to contamination of the holographic projections. Furthermore, OLAS produces ringing artifacts around high-contrast edges and in the periphery leading to image quality degradation. The proposed hogel-free holography suppresses severe rinigng and reconstruction noise, achieving significantly higher spatial resolution, light efficiency, and contrast than all of the prior methods.

The full color results as seen in Figures 12 and 13 confirm the improvements as seen in simulations. Our method reduces ringing and reconstruction noise present in existing methods and overcomes the fundamental spatio-angular resolution tradeoff. As a result, fine details such as the blades of grass are accurately captured in 3D holograms generated using HFH, along with their associated defocus effects as the user (camera) focus changes. On the other hand, it can be clearly seen that the previous methods which relied on single-step propagation and heuristic encoding such as double-phase amplitude encoding sacrifice spatial resolution. For example, observe that the cattails near the pond are very well reconstructed using hogel-free holography, whereas a significant spatial resolution drop can be observed with the other methods. While some artifacts do remain in our hardware captures due to imperfections of the SLM, uncalibrated phase look-up tables, and DC unmodulated light, these artifacts can be eliminated by tightly calibrating the hardware with recent hardware-in-the-loop optimization schemes [Chakravarthula et al. 2020; Peng et al. 2020]. We note that previous methods could not be remedied with hardware-in-the-loop refinement due to their restriction on both amplitude and phase, and non-iterative nature of hologram generation; our hogel-free holography is the first to bridge this gap.

6.2 Depth of Field Effects

We validate the ability of our method to produce high-quality true 3D holograms by measuring and evaluating the defocus effects of holographic projections. To ensure tight focusing on the projected 3D scene, we first display a calibration spokes wheel image at different focus distances to adjust the camera focus to the desired distance. The true 3D hologram projection is then imaged with the calibrated camera focus, with the camera settings kept constant across all focus distances. Our method produces appropriate defocus effects as seen from Figure 12. Specifically, observe the defocus effects between the grass blades in the foreground and the white tree in the background for 3D scenes at the top. Similarly, the cattails at the pond go out of focus as the camera is changed from far focus to near focus. In Figure 13, we evaluate the defocus effects on a more challenging scene where the depth map is insufficient to describe the depth of the scene, and much of the 3D scene is visible due to the reflection from a mirror. Our method is able to produce appropriate defocus effects when the camera is focused on objects reflected from the mirror.

6.3 Parallax and Occlusion Effects

We study the parallax and occlusion effects produced by holograms optimized using hogel-free holography. To capture these view-dependent effects, an aperture is placed on a translating stage in the Fourier plane of of our setup to filter the angular views. We use a camera viewpoint baseline of 4 mm with an identical sized aperture in the Fourier plane. As the eye-pupil is typically about 4 mm [Kahneman and Beatty 1966], this baseline supports intra-ocular occlusions. Figures 14 and 15 validate that our method produces true 3D holograms with accurate parallax and occlusion effects. In Figure 14, observe the white tree in the background of the leftmost scene which is visible in the left view but nearly disappears in the right view. Similarly, observe the blades of grass that appear in the left view but disappear in the right view. The proposed method produces a wavefield that continuously encodes the angular views of the light field. Therefore, Figure 15 shows similar parallax effects for the trees reflected in the mirror despite inaccurate cues from the depth map (also see Figure 9). Specifically, observe how more of the blue sky is revealed in one view than the other. We refer to the Supplementary Video for a dynamic illustration of the parallax effects.

Comparison to Tensor Holography. The tensor holography [Shi et al. 2021] neural network works only for a specific HoloEye Pluto SLM which has a pixel pitch of \( 8 \,{µ}{\rm m} \), significantly different compared to our HoloEye Leto SLM with a \( 6.4 \,{µ}{\rm m} \) pixel pitch. As a result, while we provided comparisons with tensor holography in simulation (see Section 5.2), we are unable to compare the method on a real holographic display. However, we note that our method significantly outperforms tensor holography as validated in simulations. The 3D volume supported by tensor holography is within a modest 6 mm depth range which is magnified by an eyepiece, whereas our continuous volume optimization method supports significantly larger depth ranges. Moreover, due to its dependence on a single RGB-D image, an insufficient depth map results in failure of producing true 3D holograms, such as the case described in Figure 9. Additionally, due to inaccurate and insufficient modeling of occlusions, tensor holography does not support parallax, and results in visible ringing artifacts and light leakage at depth discontinuities (see Figure 10).

Fig. 10.

View Figure

Fig. 11.

View Figure

Fig. 12.

View Figure

Fig. 13.

View Figure

Fig. 14.

View Figure

Fig. 15.

View Figure