2.1 Differences between CALIOP (CALIPSO) and ATLID (EarthCARE) spaceborne lidars

CALIOP, a two-wavelength polarization-sensitive near-nadir-viewing lidar, provides high-resolution vertical profiles of aerosols and clouds (Winker et al., 2010). Its initial orbital altitude was 705 km (now 688 km to match that of CloudSat), and its orbit is inclined at 98.05^∘. The lidar overpasses the Equator at 01:30 and 13:30 LST (local solar time). It uses three receiver channels: one channel measuring the 1064 nm backscatter intensity and two channels measuring orthogonally polarized components of the 532 nm backscattered signal. Cloud and aerosol layers are detected by comparing the measured 532 nm signal return with the return expected from a molecular atmosphere (see the definitions later). The other instrumental parameters of this lidar are described in Table 1 (see also Fig. 1 of Hunt et al., 2009, for a block diagram of CALIOP).

The goals of the EarthCARE mission are “to retrieve vertical profiles of clouds and aerosols, and the characteristics of their radiative and microphysical properties to determine flux gradients within the atmosphere and fluxes at the Earth's surface, as well as to measure directly the fluxes at the top of the atmosphere and also to clarify the processes involved in aerosol-cloud and cloud-precipitation-convection interactions” (Héliere et al., 2017; Illingworth et al., 2015). The ATLID instrument on board the EarthCARE satellite will measure the attenuated atmospheric backscatter with a vertical resolution of ∼100 and ∼500 m in the altitude ranges of 0–20 km and 20–40 km, respectively. ATLID is a polarization-sensitive, high-spectral-resolution lidar (HSRL), which can separate the thermally broadened molecular backscatter (Rayleigh) from the unbroadened backscatter from atmospheric particles (Mie) (Durand et al., 2007; see also Fig. 2 of do Carmo et al., 2021). This helps ATLID retrieve extinction and backscatter vertical profiles without assuming the extinction-to-backscatter ratio (as in CALIOP retrievals), which is poorly known, especially for aerosols (e.g., Rogers et al., 2014).

Table 1Specifications of the CALIOP and ATLID spaceborne lidars considered in this article. We gathered specifications from Hunt et al. (2009) for CALIOP and from do Carmo et al. (2021) for ATLID. PMT: photomultiplier tube, APD: avalanche photodiode.

^a The nominal orbit altitude at launch was 705 km but was lowered to 688 km in September 2018 to maintain formation flying with CloudSat. ^b The original pulse repetition frequency of ATLID laser is 51 Hz at the energy of 35 mJ per pulse, but the measurements will be doubled on board the satellite (do Carmo et al., 2021), so one can consider the effective frequency and energy per pulse to be equal to 25.5 and 70 mJ, respectively. ^c The solar-filter bandwidth of ATLID is 0.71 nm, but the transmission function of the Mie channel is approximately half of that, so one should calculate the solar noise in this channel with a narrower effective filter width.

Download Print Version | Download XLSX

When considering signal quality and performance, some parameters are in favor of CALIOP (telescope diameter, energy per pulse, solar-filter bandwidth), whereas others favor ATLID (altitude, noise level). In the next section, we show how these differences affect the detectability of clouds. We excluded the multiple scattering coefficient from the table since it is an important and complex parameter of lidar instrument which depends on its several parameters. Instead, we discuss it in a dedicated paragraph below.

2.2 Lidar equation

The formalism used in this work was described in Feofilov et al. (2022). In this section, we repeat only the basic definitions needed for understanding the material presented below. Since we will discuss both conventional (non-HSRL) and HSRL lidars, we will introduce necessary quantities in parallel and label them correspondingly: the molecular, particulate, and total components will get the indices “mol”, “part”, and “tot”, respectively.

An atmospheric lidar sends a brief pulse of laser radiation towards the atmosphere. The lidar optics collect the backscattered photons and drive them to a detector. The detected signal is time-resolved: supposing each photon traveled straight forward and back, each time bin corresponds to a fixed distance from the lidar to the atmospheric layer where backscattering occurred. The propagation of laser light through the atmosphere and backwards to the detector is described by the following lidar equation:

where ATB stands for attenuated total backscatter (m⁻¹ sr⁻¹), β_mol(λ,z) and β_part(λ,z) are the wavelength-dependent molecular and particulate backscatter coefficients (m⁻¹ sr⁻¹), α_mol(λ,z) and α_part(λ,z) are the extinction coefficients (m⁻¹), Z_sat is the altitude of the satellite, λ is the wavelength, and η is a multiple scattering coefficient (e.g., Platt, 1973; Garnier et al., 2015; Donovan, 2016).

For the HSRL lidar, one can write similar equations for the attenuated radiance backscattered from atmospheric particles and molecules (APBs and AMBs), respectively:

For cloud definition, we will also need to define the attenuated molecular backscatter for clear-sky conditions:

The physical meaning of η in Eqs. (1)–(3) is an increase in the number of photons remaining in the lidar receiver field of view (FOV) besides the ones directly backscattered by the layer, and its value depends on the type of scattering media, FOV of the telescope, and laser beam divergence. The typical value of η varies between 0.5 and 0.8 for commonly used lidars (Chiriaco et al., 2006; Chepfer et al., 2008, 2013; Garnier et al., 2015; Donovan, 2016; see also Appendix B of Reverdy et al., 2015). Setting η to 1 means no multiple scattering and would correspond to an infinitely narrow FOV telescope combined with an infinitely small laser beam divergence. In CALIOP cloud products up to version 3, the η was set to 0.6 for ice clouds, whereas for version 4.10 a temperature-dependent coefficient was used, which varied in between 0.46 and 0.78 (Young et al., 2018). For water clouds, the η values are derived from the relationship developed in Hu et al. (2006) (also see Table 4 in Young et al., 2018). A detailed modeling of η for different cloud types observed by CALIOP and ATLID (Shcherbakov et al., 2022) shows that η depends on the cloud thickness and type and that the ATLID values are somewhat higher than those of CALIOP. Based on these works, we set a fixed value of η to 0.6 for CALIOP and to 0.75 for ATLID. This is an approximation, and a more complex approach might be required for processing real data, but our tests show that the conclusions of the present work do not change if we vary η within ±0.1 either for CALIOP or for ATLID (but not for both).

2.3 Cloud detection and cloud variables

To characterize the scattering properties of the atmosphere, it would be convenient to use some ratio of attenuated backscatter values (Eqs. 1–4), which would have a clear physical interpretation. Due to attenuation of AMBs below the clouds, using it in the denominator is counterproductive, so the ATB_mol(λ,z) is used instead, and the scattering ratio (SR) is defined as

Considering a single-pulse profile measurement, we define a layer as cloudy if the following two conditions are met:

The second condition in Eq. (6) comes from the fact that the molecular backscatter in the upper troposphere is weak and the fluctuation in ATB might cause a false cloud detection if only SR is used. With the second condition, the cloud detection is more robust. This definition is used in CALIPSO-GOCCP (e.g., Chepfer et al., 2008, 2010, 2013), and we suggest keeping it for other lidars to ensure consistency between cloud products as discussed later.

In application to ATLID, this will mean using the values recalculated to 532 nm of ATB, which will be estimated from Eq. (1): β_part(355 nm,z) and α_part(355 nm,z) retrieved from the measurements (Eqs. 2 and 3) and β_mol(532 nm,z) and α_mol(532 nm,z) retrieved or estimated from pressure-temperature profiles from reanalysis. In the numerical experiment below, we calculated ATB${}_{\mathrm{mol}}\left(\mathrm{532}\mathrm{nm},z\right)={\mathit{\beta }}_{\mathrm{mol}}\left(\mathrm{532}\mathrm{nm},z\right)\times e^{-\mathrm{2}{\int }_{Z_{\mathrm{sat}}}^z{\mathit{\alpha }}_{\mathrm{mol}}\left(\mathrm{532},z^{\prime }\right)\mathrm{d}{z}^{\prime }}$ using the available pressure-temperature profiles and the formalism provided in Feofilov et al. (2022). Here, we reproduce Eq. (8) of this paper:

In this conversion, we assume that the spectral dependence of particulate backscatter (β_part(λ) and α_part(λ)) is weak at the wavelengths used in this study. In Beyerle et al. (2001) it is stated that this is generally true for cirrus. In Voudouri et al. (2020), the values at two wavelengths agree within (relatively large) error bars. Therefore, we do not attempt to compensate for the spectral dependence. The only area where we noticed that this approach does not work for real data is the polar stratospheric region, where a direct application of Eq. (7) leads to an overestimation of polar stratospheric clouds (PSCs) (Fig. 8b of Feofilov et al., 2022).

For the cloud properties, we use the same variables as in CALIPSO-GOCCP (Chepfer et al., 2010): cloud fraction CF(z), opaque cloud cover C_opaque, and opaque cloud height Z_opaque. If a given atmospheric layer was observed multiple times or if it was sampled vertically at several points, we define the cloud fraction profile CF(z) in a usual way:

where N_cld(z) is the number of times the conditions of Eq. (6) is met and N_tot(z) is the total number of measurements in this layer. The opaque cloud cover C_opaque is used in long time series and is defined over the 2^∘ × 2^∘ latitude–longitude gridded data as follows:

where the first condition triggers the opaque cloud detection (Guzman et al., 2017), N_{opaque_prof} is the number of vertical profiles for which an attenuation corresponding to a presence of opaque cloud was found, and N_{total_prof} is the total number of measurements in a 2^∘ × 2^∘ grid box. For an individual lidar profile, Z_opaque corresponds to an altitude of full attenuation of backscattered signal, whereas for gridded data, Z_opaque is an opaque-cloud-cover-weighted sum (Guzman et al., 2017).

3.1 Cloud-generating model

The 3DCLOUD model (Szczap et al., 2014) generates three-dimensional (3D) spatial structures of stratocumulus and fair-weather cumulus and cirrus that share some statistical properties observed in real clouds such as the inhomogeneity parameter ρ (standard deviation normalized by the mean of the water content) and the Fourier spectral slope $\widehat{\mathit{\beta }}$ close to $-\mathrm{5}/\mathrm{3}$ between the smallest scale of the simulation to the outer scale L_out (where the spectrum becomes more flat). We assume that water content follows a gamma distribution. 3DCLOUD_V2 presented in Alkasem et al. (2017) is based on wavelet framework instead of the Fourier framework. First, 3DCLOUD assimilates meteorological profiles (humidity, pressure, temperature, and wind velocity) and solves drastically simplified basic atmospheric equations in order to simulate 3D water content. Second, the Fourier filtering method is used to constrain the intensity of mean water content, ρ, $\widehat{\mathit{\beta }}$, and L_out, which are values provided by the user (Hogan and Illingworth, 2003; Kärcher et al., 2018).

Conditions of simulations to generate the stratocumulus in this study (see Fig. 1) are identical to those used in Szczap et al. (2014) for the DYCOMS2-RF01 case (the first research flight of the second Dynamics and Chemistry of Marine Stratocumulus) for the marine stratocumulus regime (Stevens et al., 2005). We have only changed the number of voxels in the x, y, and z directions to $N_x=N_y=\mathrm{1000}$ and N_z=50, respectively. The corresponding spatial resolutions were set to ${\mathrm{\Delta }}_x={\mathrm{\Delta }}_y=\mathrm{100}$ m and Δ_z=24 m, respectively. The vertical extension of the simulated area is still L_z=1200 m, but the horizontal extensions for this study are $L_x=L_y=\mathrm{100}$ km.

Figure 1Examples of the stratocumulus generated with 3DCLOUD: (a) 2D ice water path and (b) its volume rendering.

Download

If the number of voxels is large, the 3DCLOUD and 3DCLOUD_V2 are very time-consuming (see Table 1 in Szczap et al., 2014) and cannot assimilate the fractional coverage for cirrus clouds. Therefore, we developed 3DCLOUD_V3 to overcome these two drawbacks for a cirrus cloud. This model will be published elsewhere. Here, we present only an outline of the 3DCLOUD_V3 algorithm.

To increase the calculation speed in 3DCLOUD_V3, we generate clouds using modified statistical tools developed as part of stage 2 of 3DCLOUD. The first stage of 3DCLOUD (i.e., the step of solving simplified basic atmospheric equations, which is very time-consuming) is no longer carried out in 3DCLOUD_V3. Thereby, 3DCLOUD_V3 can be seen as a purely stochastic cirrus cloud generator. The user has to provide, in addition to N_x, N_y, N_z, Δ_x, Δ_y, and Δ_z, the mean ice water path (IWP); L_out; the shape of the vertical profile of ice water content (IWC), ρ(z), and $\widehat{\mathit{\beta }}\left(z\right)$ and of horizontal wind velocity components u(z) and v(z); and finally the cloud fraction CF. The shape of the vertical profile of IWC can also be stipulated (rectangular, upper triangle, lower triangle, and isosceles trapezoid; Feofilov et al., 2015). The algorithm works as follows:

1. The 3D isotropic field with a Gaussian probability density function (PDF) is generated from a 3D inverse Fourier transform assuming a random phase for each Fourier amplitude and a 3D spectral energy density with 1D spectral slope $\widehat{\mathit{\beta }}$ close to $-\mathrm{5}/\mathrm{3}$ between the smallest scale of L_out.

2. The 2D Gaussian PDF is transformed into a 2D Gamma PDF at each z level, satisfying the values of IWC(z), ρ(z), and $\widehat{\mathit{\beta }}\left(z\right)$.

3. There is horizontal displacement, at each z level, of 2D IWC (to simulate fall streaks) computed from u(z) and v(z), based on the model of sedimentation proposed by Hogan and Kew (2005). In 3DCLOUD_V3, the user can choose the value of the sedimentation velocity: either constant or as a function of IWC (see formula in Fig. 12 in Heymsfield et al., 2017). Alternatively, the wind velocity vertical profile can be computed from a constant value of the vertical wind shear prescribed by the user; in this case, the user also has to provide the “generated-level height” as explained in Hogan and Kew (2005).

4. The vertical profile of the cloud cover is iteratively modified in order to obtain the CF value prescribed by the user.

Figure 2 demonstrates the examples of 2D IWP and the 3D IWC volume rendering of the cirrus generated with 3DCLOUD_V3, where $N_x=N_y=\mathrm{1000}$, N_z=100, ${\mathrm{\Delta }}_x={\mathrm{\Delta }}_y=\mathrm{100}$ m, and Δ_z=20 m. The mean IWP is set to 1 g m⁻². The IWC vertical profile shape is “rectangular”. The geometric depth is 2 km. The outer scale is L_out=20 km. We set the constant vertical wind shear to 5 m s⁻¹ km⁻¹ in the x and y directions, and the generated-level height is 400 m under the cloud top. The inhomogeneity parameter of IWC is ρ=0.4. The spectral slope β is equal to $-\mathrm{5}/\mathrm{3}$. Figure 2c shows the gamma-like PDF of the IWC (we ignored null values), and Fig. 2d shows the mean power spectra of IWP (and IWC) along x and y directions (and z direction), with the 1D spectral slope close to −2.0 (−1.3) between L_out=20 km and the finest spatial resolution. As expected, values of the spectral slope of IWP are smaller than those of IWC (i.e., the IWP signal is “smoother” than the IWC signal) because IWP is the vertically integral quantity of IWC. One can note that the IWC spectral slope is slightly smaller than the prescribed theoretical value $\widehat{\mathit{\beta }}=-\mathrm{5}/\mathrm{3}$ because of the many null values of the IWC; we plan to remove this bias in the final version of 3DCLOUD_V3.

Figure 2Examples of cirrus generated with 3DCLOUD_V3: (a) ice water path and (b) its volume rendering; (c) IWC PDF; (d) mean 1D power spectrum of IWP (red curves) and of IWC (blue curve) following the x, y and z direction (solid, dash–dot and dashed line, respectively). A theoretical power spectrum with spectral slope $\widehat{\mathit{\beta }}=-\mathrm{5}/\mathrm{3}$ is added (dashed black line). A dotted vertical black line indicates the outer scale L_out=20 km.

Download

3.2 Numerical chain to simulate cloud observations by CALIOP and ATLID at high resolution

3.2.1 Creating pseudo orbits

We performed the following numerical experiment, outlined in the flowchart in Fig. 3. First, we created a gridded global atmosphere from the output of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SM) atmosphere model version 1 (EAMv1; Rasch et al., 2019) for the conditions of autumn equinox in the Northern Hemisphere. Since we wanted to address both high- and low-level cloud detection, we picked up only the tropical part of the orbit between 5^∘ S and 5^∘ N and used this data as a set of smooth “background” profiles. Since this model does not provide the small-scale variability needed for our experiment, we used the subgrid model described in Sect. 3.1, which generates realistic cloud profiles at a grid comparable or finer than the distance between two consecutive footprints of studied lidars. To address the most challenging observation conditions, we picked up two cloud types: (1) thin cirrus clouds with optical depths (τ) of about 0.03–0.1 per layer (Sassen and Comstock, 2001) and (2) stratocumulus clouds with their high horizontal variability and large optical depths (up to 30 but with about one-third of semi-transparent clouds). These clouds were simulated using an updated 3DCLOUD_V3 model (see Sect. 3.3) and provided as gridded sets of ice water content (IWC) and liquid water content (LWC) values for cirrus and stratocumulus clouds, respectively. We do not consider another challenging case, a thin-cloud layer above a highly reflective cloud, but the daytime noise estimated for the stratocumulus scene will give an idea of what background noise will be interfering with the useful cloud signal in this case.

Figure 3Flowchart explaining the numerical experiment on comparing clouds retrieved from CALIOP and ATLID observations. Green boxes list the input and output data. Black text between boxes describes actions performed on each dataset. Blue text in the boxes marks the datasets used in the estimation. White text in square brackets in the boxes indicates horizontal (H) and vertical (V) resolutions of the datasets. Note that the ATLID SR^′ values are estimated at 532 nm (see Sect. 2.3).

Download

These gridded sets were converted to pseudo orbits by slicing them along the diagonal lines and arranging the slices into “lidar curtains”, each comprising 20 000 individual profiles and split into daytime and nighttime parts with 10 000 profiles each. This way we got almost seamless cloud distributions, which followed the variability prescribed by the 3DCLOUD_V3 model and at the same time resembled parts of real lidar orbits. We show the most representative parts of these pseudo orbits in Figs. 4 and 5 for cirrus and stratocumulus clouds, respectively, and we discuss them below.

With these two datasets covering both the daytime and the nighttime scenes, we performed a full series of simulations, explained in Fig. 3. Namely, we fed the high-resolution atmospheric inputs described above to the CALIOP and ATLID simulators (Chepfer et al., 2008; Reverdy et al., 2015) included in the Cloud Feedback Model Intercomparison Project Observational Simulator Package version 2 (COSP2) simulator (Swales et al., 2018). These simulators do not account for instrumental noise effects, so their outputs were processed by a third part of the simulation chain (Fig. 3), which estimates noise and its propagation in the lidar system.

3.2.2 Estimating lidar signals and noise

As mentioned above, the outputs of COSP2 simulator are the noise-free APB (355 nm,z) and AMB (355 nm,z) profiles for ATLID and noise-free ATB (532 nm,z) profiles for CALIOP, both calculated at a horizontal resolution of 300 m and vertical resolution of 20 m. To estimate the noise for these profiles and to propagate it further to SR (532 nm,z) for CALIOP and to recalculated SR${}^{\prime }\left(\mathrm{532}\mathrm{nm},z\right)$ for ATLID, we calculated the signals from scratch using the corresponding instrumental parameters introducing measurement noise as follows.

We start with a laser emission and estimate the number of emitted photons per sounding pulse, measured at the output of the sounding unit:

$$\begin{array}{}\text{(10)}& N_{\mathrm{em}}={\displaystyle \frac{E_{\mathrm{pulse}}}{hc/\mathit{\lambda }}}\times \mathit{\varsigma },\end{array}$$

where E_pulse is the energy per laser pulse, h is Planck's constant, c is the speed of light, λ is the wavelength, and ς is an effective coefficient of optical throughput of the emission path of the lidar. In the present work, it is assumed that there is no optical loss in the emission path and ς is equal to 1. The numerical solution of the Eqs. (1)–(3) yields the number of photons per range gate (t_it_i+Δt_i), which we will denote as t_i, coming through the CALIOP receiver or through the ATLID receiver before splitting into molecular and particulate components in the HSRL module (compare to Eqs. 2 and 3):

$$\begin{array}{}\text{(11)}& {\displaystyle }{N}_{\mathrm{rec}}^{\mathrm{tot}}\left(t_i\right)=N_{\mathrm{rec}}^{\mathrm{mol}}\left(t_i\right)+N_{\mathrm{rec}}^{\mathrm{par}}\left(t_i\right),\text{(12)}& {\displaystyle }\begin{array}{rl}{N}_{\mathrm{rec}}^{\mathrm{mol}}\left(t_i\right)& =N_{\mathrm{em}}\times {\mathit{\beta }}_{\mathrm{mol}}\left(i\right)\times \mathrm{\Delta }{z}_i\times \mathrm{\Omega }\left(z_i\right)\times {\mathit{\xi }}_{\mathrm{rec}}\\ & \times e^{-\mathrm{2}{\sum }_{j=\mathrm{0}}^i({\mathit{\alpha }}_{\mathrm{mol}}\left(j\right)+\mathit{\eta }{\mathit{\alpha }}_{\mathrm{part}}(j\left)\right)\mathrm{\Delta }{z}_j},\end{array}\end{array}$$

$$\begin{array}{}\text{(13)}& {\displaystyle }\begin{array}{rl}{N}_{\mathrm{rec}}^{\mathrm{par}}\left(t_i\right)& =N_{\mathrm{em}}\times {\mathit{\beta }}_{\mathrm{par}}\left(i\right)\times \mathrm{\Delta }{z}_i\times \mathrm{\Omega }\left(z_i\right)\times {\mathit{\xi }}_{\mathrm{rec}}\\ & \times e^{-\mathrm{2}{\sum }_{j=\mathrm{0}}^i({\mathit{\alpha }}_{\mathrm{mol}}\left(j\right)+\mathit{\eta }{\mathit{\alpha }}_{\mathrm{part}}(j\left)\right)\mathrm{\Delta }{z}_j},\end{array}\text{(14)}& {\displaystyle }\mathrm{\Omega }\left(z_i\right)=\mathit{\pi }\times {\displaystyle \frac{(d_{\mathrm{tel}}/\mathrm{2}{)}^{\mathrm{2}}}{z_i^{\mathrm{2}}}},\end{array}$$

where $N_{\mathrm{rec}}^{\mathrm{mol}}\left(t_i\right)\mathrm{and}{\mathrm{N}}_{\mathrm{rec}}^{\mathrm{par}}\left(t_i\right)$ are the photons backscattered by molecules and particles, respectively; Ω(z_i) is an altitude-dependent solid angle with z_i corresponding to the time of flight t_i between the satellite and the measured layer i; and ξ_rec is the receiver's transmission.

Figure 4Example of cirrus cloud input data from the 3DCLOUD model used in the simulation: (a) ice water content (IWC), with night corresponding to one piece of the orbit; (b) IWC, with day corresponding to another piece of the orbit; (c) accumulated optical depth starting from the cloud top at night; (d) same as (c) during the day; (e) cloud mask at night; and (f) cloud mask during the day. We set the cloud mask to 1 whenever IWC > 0. The cloud masks presented here are called the “reference dataset” in the rest of the paper.

Download

Figure 5Same as Fig. 4 but for stratocumulus cloud scenes. Note the color scale difference between Figs. 4 and 5.

Download

For an HSRL lidar, the molecular and particulate components are supposed to be registered individually, but this separation is not ideal because of the crosstalk between the channels: a part of molecular backscatter comes at the same wavelength as the original laser radiance and it “contaminates” the particulate channel, which is centered at this wavelength. Overall, the HSRL system is characterized by four crosstalk coefficients, C_mm, C_pp, C_mp, and C_pm. The first two show a contribution of the molecular and particulate channels to themselves, and in the ideal HSRL they should be equal to 1. The second pair shows how much energy “leaks” from a molecular channel to a particulate one and vice versa. In an ideal HSRL, these coefficients would be equal to 0. In the operational retrieval, these coefficients will be determined through a continuous calibration procedure performed on the orbit. For this exercise, we estimated these coefficients from the Fabry–Pérot interferometer spectral curves (Cheng et al., 2013): C_mm=0.815, C_pp=0.60, C_mp=0.185, and C_pm=0.40. In the case of a non-ideal HSRL, the number of photoelectrons produced by each detector per range gate is as follows:

$$\begin{array}{}\text{(15)}& {\displaystyle }{N}_{\det }^{\mathrm{mol}}\left(t_i\right)=\mathit{\gamma }\times {\mathit{\xi }}_{\mathrm{rec}}\times \left(N_{\mathrm{rec}}^{\mathrm{mol}}\left(t_i\right)\times c_{\mathrm{mm}}+N_{\mathrm{rec}}^{\mathrm{par}}\left(t_i\right)\times c_{\mathrm{pm}}\right),\text{(16)}& {\displaystyle }{N}_{\det }^{\mathrm{par}}\left(t_i\right)=\mathit{\gamma }\times {\mathit{\xi }}_{\mathrm{rec}}\times \left(N_{\mathrm{rec}}^{\mathrm{mol}}\left(t_i\right)\times c_{\mathrm{mp}}+N_{\mathrm{rec}}^{\mathrm{par}}\left(t_i\right)\times c_{\mathrm{pp}}\right),\end{array}$$

where γ and ξ_rec are the detector's quantum efficiency and transmittance of the optical path, respectively. To come back to “pure” $N_{\mathrm{rec}}^{\mathrm{mol}}\left(t_i\right)$ and $N_{\mathrm{rec}}^{\mathrm{par}}\left(t_i\right)$ used in the retrieval, one has to solve this system:

$$\begin{array}{}\text{(17)}& {\displaystyle }{N}_{\mathrm{rec}}^{\mathrm{mol}}\left(t_i\right)=N_{\det }^{\mathrm{mol}}\left(t_i\right)\times k_{\mathrm{a}}+N_{\det }^{\mathrm{par}}\left(t_i\right)\times k_{\mathrm{b}},\text{(18)}& {\displaystyle }{N}_{\mathrm{rec}}^{\mathrm{par}}\left(t_i\right)=N_{\det }^{\mathrm{mol}}\left(t_i\right)\times k_{\mathrm{c}}+N_{\det }^{\mathrm{par}}\left(t_i\right)\times k_{\mathrm{d}},\text{(19)}& {\displaystyle }\begin{array}{rl}{k}_{\mathrm{a}}& ={\displaystyle \frac{c_{\mathrm{pp}}}{\mathit{\kappa }}};k_{\mathrm{b}}={\displaystyle \frac{-c_{\mathrm{pm}}}{\mathit{\kappa }}};k_{\mathrm{c}}={\displaystyle \frac{-c_{\mathrm{mp}}}{\mathit{\kappa }}};k_{\mathrm{d}}={\displaystyle \frac{c_{\mathrm{mm}}}{\mathit{\kappa }}};\mathit{\kappa }\\ & =\mathit{\gamma }\times {\mathit{\xi }}_{\mathrm{rec}}\times \left(c_{\mathrm{mm}}{c}_{\mathrm{pp}}-c_{\mathrm{pm}}{c}_{\mathrm{mp}}\right).\end{array}\end{array}$$

Besides the components related to atmospheric backscatter properties, the $N_{\det }^{\mathrm{mol}}\left(t_i\right)$ and $N_{\det }^{\mathrm{par}}\left(t_i\right)$ are affected by “parasite” solar backscattered photons during the daytime, which are not correlated with the laser shots. To estimate the solar background add-on to $N_{\det }^{\mathrm{mol}}\left(t_i\right)$ and $N_{\det }^{\mathrm{par}}\left(t_i\right)$, one has to solve the radiative transfer equation for the radiation emitted by the Sun, backscattered by air and particles in the atmosphere and by the surface in the direction of the spaceborne lidar, and attenuated by the atmospheric layers:

$$\begin{array}{}\text{(20)}& N_{\mathrm{sol}}\left(t_i\right)=\mathrm{\Delta }{t}_i\times N_{\mathrm{sol}}^{\mathrm{TOA}}\left(\mathit{\lambda }\right)\times \left[R_{\mathrm{sol}}^{\mathrm{atm}}+R_{\mathrm{sol}}^{\mathrm{surf}}\right],\end{array}$$

$$\begin{array}{}\text{(21)}& \begin{array}{rl}& R_{\mathrm{sol}}^{\mathrm{atm}}=\underset{Z_{\mathrm{sat}}}{\overset{\mathrm{0}}{\int }}\left({\mathit{\beta }}_{\mathrm{m}}\left(z\right)\times {\mathit{\varphi }}_{\mathrm{m}}\left(\mathrm{SZA}\right)+{\mathit{\beta }}_{\mathrm{p}}\left(z\right)\times {\mathit{\varphi }}_{\mathrm{p}}\left(\mathrm{SZA}\right)\right)\times \cos {\left(\mathrm{SZA}\right)}^{-\mathrm{1}}\\ & \times \exp \left\{-\mathrm{2}\underset{z_{\mathrm{sat}}}{\overset{z}{\int }}\left({\mathit{\alpha }}_{\mathrm{m}}\left(z{}^{\prime }\right)+{\mathit{\alpha }}_{\mathrm{p}}\left(z{}^{\prime }\right)\right)\cos (\mathrm{SZA}{)}^{-\mathrm{1}}\mathrm{d}{z}^{\prime }\right\}\mathrm{d}z,\end{array}\end{array}$$

$$\begin{array}{}\text{(22)}& \begin{array}{rl}& R_{\mathrm{sol}}^{\mathrm{surf}}=A_{\mathrm{surf}}\times {\mathit{\varphi }}_{\mathrm{surf}}\left(\mathrm{SZA}\right)\times \exp \left\{-\underset{z_{\mathrm{sat}}}{\overset{z}{\int }}\left({\mathit{\alpha }}_{\mathrm{m}}\left(z\right)\right.\right.\\ & \left.+{\mathit{\alpha }}_{\mathrm{p}}\left(z\right)\right)\cos (\mathrm{SZA}{)}^{-\mathrm{1}}\mathrm{d}z\mathit{\}}\times \underset{Z_{\mathrm{sat}}}{\overset{\mathrm{0}}{\int }}\left({\mathit{\alpha }}_{\mathrm{m}}\left(z\right)+{\mathit{\alpha }}_{\mathrm{p}}\left(z\right)\right)\mathrm{d}z,\end{array}\end{array}$$

where $N_{\mathrm{sol}}^{\mathrm{TOA}}\left(\mathit{\lambda }\right)$ is the top-of-atmosphere solar flux at wavelength λ and for filter width Δλ; $R_{\mathrm{sol}}^{\mathrm{atm}}$ and $R_{\mathrm{sol}}^{\mathrm{surf}}$ represent the proportion of the incoming solar radiance reflected in the direction of lidar; ϕ_m(SZA), ϕ_p(SZA), and ϕ_surf(SZA) are the scatter plots for the angle between the Sun and the nadir view of lidar for molecular scattering, scattering on particles, and scattering from the surface, respectively; z_sat is the altitude of a satellite; and A_surf is the surface albedo. We assumed Lambertian scattering from the surface with albedo equal to 0.08 for ocean and 0.15 for land (arbitrary values), we used Rayleigh scattering phase function for the molecular component, and we used the geometric-optics phase function approximation for particulate scattering.

The solar photons pass through the optical system and HSRL, hit the detectors, and produce the “solar-noise photoelectrons”:

$$\begin{array}{}\text{(23)}& {\displaystyle }{N}_{\det .\mathrm{sol}.}^{\mathrm{mol}}\left(t_i\right)=\mathit{\gamma }\times {\mathit{\xi }}_{\mathrm{rec}}\times {\mathit{\xi }}_{\mathrm{mol}.\mathrm{sol}.}\times N_{\mathrm{sol}}\left(t_i\right),\text{(24)}& {\displaystyle }{N}_{\det .\mathrm{sol}.}^{\mathrm{par}}\left(t_i\right)=\mathit{\gamma }\times {\mathit{\xi }}_{\mathrm{rec}}\times {\mathit{\xi }}_{\mathrm{par}.\mathrm{sol}}\times N_{\mathrm{sol}}\left(t_i\right),\end{array}$$

where ${\mathit{\xi }}_{\mathrm{mol}.\mathrm{sol}.}$ and ${\mathit{\xi }}_{\mathrm{par}.\mathrm{sol}.}$ represent the convolution of the solar-filter spectral curve with the interferometric spectral curve for a given channel (see the comment on solar-filter bandwidth in Table 1). In addition to solar noise, there is always a dark current of the detector (N_dark) and readout noise (RON) which are added to the signal. Since the $N_{\det .\mathrm{sol}.}^{\mathrm{mol}}\left(t_i\right)$ and $N_{\det .\mathrm{sol}.}^{\mathrm{par}}\left(t_i\right)$, N_dark, and RON are registered along with $N_{\det }^{\mathrm{mol}}\left(t_i\right)$ and $N_{\det }^{\mathrm{par}}\left(t_i\right)$ during daytime, they enter Eqs. (15) and (16) and affect the retrieval. For the non-HSRL lidar:

$$\begin{array}{}\text{(25)}& N_{\det .\mathrm{sol}.}\left(t_i\right)=\mathit{\gamma }\times {\mathit{\xi }}_{\mathrm{rec}}\times {\mathit{\xi }}_{\mathrm{tot}.\mathrm{sol}.}\times N_{\mathrm{sol}}\left(t_i\right),\end{array}$$

where γ and ξ_rec stand for the corresponding parameters of non-HSRL lidar and ${\mathit{\xi }}_{\mathrm{tot}.\mathrm{sol}.}$ represents the transmission coefficient of a solar rejection filter, which is equal to a ratio of an integral of the spectral transmission curve of the filter to a full spectral width of the filter. A quick back-of-the-envelope estimate of the ratio of solar photons coming to the particulate detector of ATLID to the number of solar photons reaching the surface of CALIOP's detector per same sampling interval is

$$\begin{array}{}\text{(26)}& \begin{array}{rl}& {\displaystyle \frac{N_{\mathrm{sol}}^{\mathrm{TOA}}\left(\mathrm{355}\mathrm{nm}\right)\times d_{\mathrm{tel}.\mathrm{ATLID}}^{\mathrm{2}}\times d_{\mathrm{fp}.\mathrm{ATLID}}^{\mathrm{2}}\times Z_{\mathrm{CALIOP}}^{\mathrm{2}}\times {\mathit{\xi }}_{\mathrm{rec}.\mathrm{ATLID}}\times {\mathit{\xi }}_{\mathrm{par}.\mathrm{sol}.}}{N_{\mathrm{sol}}^{\mathrm{TOA}}\left(\mathrm{532}\mathrm{nm}\right)\times d_{\mathrm{tel}.\mathrm{CALIOP}}^{\mathrm{2}}\times d_{\mathrm{fp}.\mathrm{CALIOP}}^{\mathrm{2}}\times Z_{\mathrm{ATLID}}^{\mathrm{2}}\times {\mathit{\xi }}_{\mathrm{rec}.\mathrm{CALIOP}}\times {\mathit{\xi }}_{\mathrm{tot}.\mathrm{sol}.}}}=\\ & {\displaystyle \frac{(\mathrm{1162.8}\times \mathrm{355})\times {\mathrm{0.6}}^{\mathrm{2}}\times {\mathrm{690}}^{\mathrm{2}}\times {\mathrm{29}}^{\mathrm{2}}\times \mathrm{0.62}\times (\mathrm{0.35}\times \mathrm{0.87})}{(\mathrm{1900.0}\times \mathrm{532})\times {\mathrm{1.0}}^{\mathrm{2}}\times {\mathrm{393}}^{\mathrm{2}}\times {\mathrm{90}}^{\mathrm{2}}\mathrm{0.67}\times (\mathrm{0.04}\times \mathrm{0.85})}}=\mathrm{0.38}.\end{array}\end{array}$$

So, at first sight, the ATLID retrieval should be less solar-contaminated than CALIOP. But, this ratio alone is not enough for such a conclusion because the solar photons should be compared to the useful signal. Below, we show the results of simulations for two atmospheric scenarios which consider two-way radiative transfer both for solar radiance and for lidar sounding radiance and add the noise of the remaining detection path.

Now, when all the components of the signal are known, we can estimate the daytime and nighttime signal and noise and propagate them to the retrieved parameters. It is important to mention that the instruments compared in this work use the detectors of different types. Namely, the CALIOP lidar uses a photomultiplier tube (PMT), whereas the ATLID lidar detects the backscatter with the help of a charge-coupled device (CCD). Besides different characteristics like gain or dark current (see Table 1), these detectors are not the same in terms of applicable noise statistics (Liu et al., 2006). Even though the incoming photon flux distributions for both instruments are Poisson, the photoelectrons produced by the PMT do not follow a strict Poisson distribution. It is known that for Poisson-distributed signals, a one-to-one relationship exists between the mean and the variance of the photocurrent. As Liu et al. (2006) show, the mean and the variance of the PMT photocurrent are also proportional but not one to one, and the corresponding noise scale factor (NSF) has to be applied to estimate random errors for lidar systems using PMTs or avalanche detectors. The NSF is linked to an excess noise factor (ENF), but it is not equal to it. For the PMTs with identical gain factors m for each dynode, the ENF is given by (Kingston, 1978; Liu and Sugimoto, 2002)

$$\begin{array}{}\text{(27)}& \mathrm{ENF}={\displaystyle \frac{m}{m-\mathrm{1}}}.\end{array}$$

For the analog detection, the NSF in the multiplied photoelectron domain can be either calculated from the detector's ENF and gain or estimated from the solar-noise-dominated signals (Liu et al., 2006):

$$\begin{array}{}\text{(28)}& \mathrm{NSF}={\displaystyle \frac{\mathit{\sigma }\left(N_{\det }\right)}{\sqrt{\left(N_{\det }\right)}}}={\displaystyle \frac{\sqrt{\mathrm{var}\left(N_{\det }\right)}}{\sqrt{\left(N_{\det }\right)}}},\end{array}$$

where σ(N_det), var(N_det), and 〈N_det〉 are the standard deviation, variance, and mean of the signal, respectively. The NSF value provided in the CALIPSO L1 version 4.10 files is equal to 5.14. However, using this value in synthetic-noise calculations leads to an overestimation of the daytime noise, so for the calculations below we took a more conservative value of NSF =3.16. One can write the expressions for the variances of CALIOP and ATLID signals in the analog detection domain through the number of photoelectrons calculated for each channel:

$$\begin{array}{}\text{(29)}& {\displaystyle }\mathrm{var}\left(N_{\det }^{\mathrm{mol}}\right)=\mathrm{ENF}\times \left(N_{\det }^{\mathrm{mol}}+N_{\mathrm{dark}}+N_{\det .\mathrm{sol}.}^{\mathrm{mol}}\right)+\mathrm{RON},\text{(30)}& {\displaystyle }\mathrm{var}\left(N_{\det }^{\mathrm{par}}\right)=\mathrm{ENF}\times \left(N_{\det }^{\mathrm{par}}+N_{\mathrm{dark}}+N_{\det .\mathrm{sol}.}^{\mathrm{par}}\right)+\mathrm{RON},\text{(31)}& {\displaystyle }\mathrm{var}\left(N_{\det }^{\mathrm{tot}}\right)={\mathrm{NSF}}^{\mathrm{2}}\times \left(N_{\det }^{\mathrm{tot}}+N_{\mathrm{dark}}+N_{\det .\mathrm{sol}.}^{\mathrm{tot}}\right)+\mathrm{RON}.\end{array}$$

We draw the reader's attention to the fact that the detector's parameters in Eq. (31) are not equal to those in Eqs. (29) and (30) and that for real calculations one has to use the values from Table 1 or a similar source. The variances of molecular, particulate, and total incoming photon fluxes, which are finally used in the optical property retrievals, are estimated in accordance with the standard error propagation formulae applied to the equations above:

$$\begin{array}{}\text{(32)}& \begin{array}{rl}\mathrm{var}\left(\mathrm{AMB}\right)& =k_{\mathrm{a}}^{\mathrm{2}}\times \mathrm{var}\left(N_{\det }^{\mathrm{mol}}\right)+k_{\mathrm{b}}^{\mathrm{2}}\times \mathrm{var}\left(N_{\det }^{\mathrm{par}}\right)+\mathrm{2}\\ & \times k_{\mathrm{a}}\times k_{\mathrm{b}}\times \mathrm{cov}\left(N_{\det }^{\mathrm{mol}}{N}_{\det }^{\mathrm{par}}\right),\end{array}\end{array}$$

$$\begin{array}{}\text{(33)}& \begin{array}{rl}\mathrm{var}\left(\mathrm{APB}\right)& =k_{\mathrm{c}}^{\mathrm{2}}\times \mathrm{var}\left(N_{\det }^{\mathrm{mol}}\right)+k_{\mathrm{c}}^{\mathrm{2}}\times \mathrm{var}\left(N_{\det }^{\mathrm{par}}\right)\\ & +\mathrm{2}\times k_{\mathrm{c}}\times k_{\mathrm{d}}\times \mathrm{cov}\left(N_{\det }^{\mathrm{mol}}{N}_{\det }^{\mathrm{par}}\right),\end{array}\end{array}$$

$$\begin{array}{}\text{(34)}& \mathrm{var}\left(\mathrm{ATB}\right)={\displaystyle \frac{\mathrm{1}}{{\mathit{\kappa }}^{\mathrm{2}}}}\times \mathrm{var}\left(N_{\det }^{\mathrm{tot}}\right),\end{array}$$

where the $\mathrm{cov}\left(N_{\det }^{\mathrm{mol}}{N}_{\det }^{\mathrm{par}}\right)$ represents the covariance of molecular and particulate channels. This term is required because the signals in the channels are coupled through non-zero crosstalk.

When the variances are known, the original noise-free AMB, APB, and ATB profiles are modified by random noise, which is modulated by the standard deviation calculated from the variances, and the results are saved. If horizontal or vertical signal averaging is involved, the noise is scaled inversely proportional to a square root of the number of samples within the averaging interval.

3.3 Simulated ATLID and CALIPSO lidar profiles over cirrus and stratocumulus scenes

The most representative parts of pseudo orbits generated with the help of the 3D_CLOUDV3 model (Sect. 3.3) are shown in Figs. 4 and 5 for cirrus and stratocumulus clouds, respectively. We arbitrarily split the “cloud curtain” generated from the output of this model (Sect. 3.2) to “daytime” and “nighttime” by setting the solar zenith angle (SZA) to 45 and 120^∘, respectively. These values are not linked with the cloud formation mechanisms in the 3D_CLOUDV3 model; they are just needed for a second half of the simulator chain (see noise-related boxes in Fig. 3). In Fig. 4a and b, one can see a fine structure of modeled cirrus clouds. Looking at Fig. 4c and d, one can say that the clouds are optically thin. This combination makes the detection of the clouds marked in Fig. 4e and f challenging.

The stratocumulus clouds shown in Fig. 5 belong to another category of challenging observations. The clouds are closely spaced along the horizontal axis, and at the same time they are optically thick: about two-thirds of the clouds have optical thickness larger than 3 (Fig. 5c, d), but the scene also contains about one-third of semi-transparent clouds like the ones that were reported in Leahy et al. (2022). From Fig. 5c and d, one can conclude that at present there is no space-based measurement that can retrieve all the optical properties of cloud layers shown in Fig. 5e and f. Another problem of these clouds is that their horizontal averaging might bias the estimated cloud fraction (see, e.g., Fig. 4 and discussion of Feofilov et al., 2022).

In Figs. 6 and 7, we demonstrate the differences between two lidars for four scenes (cirrus/stratocumulus clouds, day/night) using the simulated backscatter signal. For the cirrus cloud scene (Fig. 6), both the ATB(532 nm,z) of CALIOP and the APB(355 nm,z) of ATLID show a detectable signal in the areas marked by a cloud mask in Fig. 6e and f. But if one defines the signal detection level as 3σ, one will see that a part of thin clouds will be missing. This is not surprising since we compare a “pure” modeled cloud with its noisy representation in the measuring system. What can be estimated from the image is the potential reliability of cloud detection from ATLID and CALIOP: according to the SNR values (Fig. 6g, h vs. c, d), the APB(355 nm,z) signal from ATLID (Fig. 6e, f) reaches higher SNR values than the ATB(532 nm,z) signal from CALIOP (Fig. 6a, b). This hints at the fact that the cloud detection from this instrument might be somewhat better than that from CALIOP and that one can lower the detection threshold and still get the cloud instead of noise. This is a subject of one of the experiments described below. As for the daytime vs. nighttime difference, we do not see a big change between the left-hand side and right-hand side panels for ATLID (Fig. 6e–h), whereas the CALIOP shows higher noise in Fig. 6b and d. We note here that the calculations were performed for the cases when only a thin cirrus cloud was present in the atmospheric column, whereas the rest of it corresponded to clear-sky conditions. In the real world, though, the second cloud layer beneath cirrus might increase the solar noise (see the right-hand side panels of Fig. 7), and this will adversely affect the thin-cloud detection, especially from the CALIOP measurements. This is explained by a larger field of view of the CALIOP lidar (see Table 1). In our exercise, we wanted to estimate the best achievable results for a given cloud scene for each instrument and to compare the lidar performances. This way, the conclusions made below for the daytime scenes refer to the minimal differences between the two instruments. As for the stratocumulus clouds (Fig. 7), both the signals and SNRs are strong for both lidars, day and night. The altitudes beneath these clouds correspond to areas without a useful signal: at these heights, the signal is already attenuated by a cloud above, and the attenuation is so strong that even the cloud base is not visible at optical wavelengths (e.g., Guzman et al., 2017). Another remarkable feature shown in this plot is higher daytime noise for CALIOP (Fig. 7b, d). Even though this high noise level does not affect the stratocumulus cloud detection itself, it might affect the aforementioned higher-level cloud detection, and, from this point of view, ATLID has an advantage over CALIOP.

Figure 6Signals and signal-to-noise ratio for the cirrus cloud scene. CALIOP: (a) ATB (532 nm, z) at night for one piece of the orbit, (b) ATB (532 nm, z) during the day for another piece of the pseudo orbit, (c) SNR at night, and (d) SNR during the day. ATLID: (e) APBs (355 nm, z) at night, (f) APBs (355 nm, z) during the day, (g) SNR at night, and (h) SNR during the day. Note that the scene contains only these clouds and a clear sky below. For the reflective clouds beneath the cirrus layer, the daytime noise will be higher (see the right-hand side panels of Fig. 7).

Download

Figure 7Same as Fig. 6 but for stratocumulus cloud scenes.

Download

Summarizing, one can say that the ATB(532 nm,z) signals of CALIOP and the APB(355 nm,z) signals of ATLID carry similar type information for the same cloud scenes, but their SNRs suggest (a) that the daytime cloud detection from ATLID should be more reliable and (b) that one can lower the detection threshold for this instrument without admixing numerous noise-triggered clouds. Let us now see how the signal quality transforms into the product quality and, in particular, to cloud detection quality.

3.4 Capability of ATLID to better detect optically thinner clouds than CALIPSO

Here, we describe the test we performed seeking to answer whether the cloud detection limits (Eq. 6) defined in Chepfer et al. (2010) could be lowered to detect thinner clouds. For this test, we followed the second half of the flowchart (Fig. 3) and calculated the SR(532 nm,z) for CALIOP and the CALIOP-like SR(532 nm,z) for ATLID (Eqs. 2–3), but we changed the cloud detection thresholds of Eq. (6) to the following ones:

Then we estimated the cloud fractions and statistical agreement with the source cloud data (Eqs. 6, 7). The threshold in the left-hand side of Eq. (36) implies that the particulate backscatter in a layer, which we call a cloudy one, is twice the molecular one.

The threshold in the right-hand side of Eq. (36) corresponds to the absolute values of ATB(532 nm,z) recalculated for $\mathrm{SR}\left(\mathrm{532}\mathrm{nm},z\right)=\mathrm{3}$ at the height of 8 km (Chepfer et al., 2010), but overall the rationale for selecting these very values is based on the SNR values levels we observed in the test simulations. Further lowering the threshold will lead to an increased number of false-positive cloud detections in ATLID.

Since the “native” CALIOP profiles are averaged over three points above 8 km, we applied an averaging procedure over ∼1 km distance to all simulated signals and repeated the analysis. To compare apples to apples in terms of signal statistics, we averaged over four CALIPSO shots and over two effective ATLID shots, yielding the actual average over 1330 and 1140 m, respectively. To reduce the number of plots, we do not show the instantaneous profiles without the averaging, but in Table 2 we provide the estimates for them (seek columns marked with “Averaged” with N).

In Fig. 8a and b, the SR(532 nm,z) has the same patterns as the ATB(532 nm,z) signals in Fig. 6a and b. But the daytime noise is more pronounced in this presentation, partially because of the chosen color scale. However, not all the noise from Fig. 8b propagates to Fig. 8d. This is because of a second condition of Eq. (36): the variations are partially filtered out by imposing a condition on the ATB(532 nm,z) signals with respect to ATB_mol(532 nm,z). Still, the daytime scene contains a lot of false detections marked by red in Fig. 8d. The overall characteristics of CALIOP cloud detection for this scene estimated over the whole simulated cloud dataset can be found in the second and sixth columns of Table 2. The bottom two lines of this table refer to the detectability of a cloud in the whole layer: if some values of the SR(532 nm,z) triggered cloud detection, we calculated the cloud fraction similar to Eq. (7) and then compared the resulting series of cloud fractions with the reference one defined from the source dataset. The “total score” line refers to the cloud detection statistics and is defined in the caption. As one can see, the strong daytime noise of CALIOP prevents the correct cloud detection, mostly due to large number of false-positive cloud detections (NO_YES). The bias and the RMS (root mean square) rows show the biggest change when passing from nighttime to daytime conditions.

Figure 8Scattering ratio and cloud detection estimated for cirrus clouds observed by CALIOP using Eq. (36): (a) scattering ratio at night, (b) cloud detection at night, (c) scattering ratio during the day, and (d) cloud detection during the day. Note the color scale difference for (a) and (c) compared to (b) and (d).

Download

Figure 9Same as Fig. 8 but for ATLID.

Download

The same analysis performed for ATLID (Fig. 9) shows less daytime noise (compare Fig. 9b to Fig. 8b), and the cloud detection quality for the clouds defined using Eq. (36) is better than that of the CALIOP (compare Fig. 9d to Fig. 8d). The corresponding columns of Table 2 tell us that for ATLID the number of false detections during day and night is approximately the same, whereas for CALIOP using the Eq. (36) for the detection dramatically increases the number of false detections during daytime. We should also stress here that the obtained result is a lower estimate because we used the scenes without underlying clouds, which could reflect more solar radiance and further contaminate the observations. For these scenes, the difference between ATLID and CALIOP will be even larger.

Table 2Cloud detection statistics for CALIOP and ATLID when the cloud definition corresponds to $\mathrm{SR}\left(\mathrm{532}\mathrm{nm},z\right)>\mathrm{3}$ and $\mathrm{ATB}\left(\mathit{\lambda },z\right)-{\mathrm{ATB}}_{\mathrm{mol}}\left(\mathit{\lambda },z\right)>\mathrm{1.5}\times {\mathrm{10}}^{-\mathrm{6}}{\mathrm{m}}^{-\mathrm{1}}{\mathrm{sr}}^{-\mathrm{1}}$ (Eq. 36). The bias and RMS values are defined for the clouds detected in the columns (see text), and we define the total score in percent as $\mathrm{100}\mathit{\%}\times (\mathrm{1}-(\mathrm{YES}\mathit{\_}\mathrm{NO}+{\mathrm{NO}}_{\mathrm{YES}})/(\mathrm{YES}\mathit{\_}\mathrm{YES}+\mathrm{NO}\mathit{\_}\mathrm{NO}\left)\right)$. Ci: cirrus clouds, Sc: stratocumulus clouds.

Download Print Version | Download XLSX

The same type plots built for the stratocumulus clouds (Figs. A1 and A2 in the Appendix A for CALIOP and ATLID, respectively) show a different picture. Strong signals and large SNRs shown in Fig. 5 help to unambiguously identify the cloud. The large fraction of underestimated clouds shown in blue in Figs. A1cd and A2cd corresponds to cloud parts below the opaque cloud top layer, which are not accessible for the instruments observing the scene from above. As with cirrus clouds, the false-detection rate is higher for CALIOP during daytime.

For CALIOP, the 1 km averages reduce the number of false detections and improve the total score for daytime simulations for cirrus. For ATLID with its lower daytime noise, the averaging procedure does not change the cloud detection quality that much. For the stratocumulus clouds, the averaging procedure is not required for ATLID since sometimes it can lead to overestimating the cloud fraction (e.g., Chepfer et al., 2008; Feofilov et al., 2022). For CALIOP, it improves the score because of suppression of sporadic noise-induced “clouds” above the real cloud layer (Fig. A1d).

Overall, the ATLID-related columns in Table 2 demonstrate more consistency between daytime and nighttime cloud amounts and reference data than the CALIOP-related ones, and ATLID daytime cloud quality is better than that of CALIOP, whereas the nighttime results are comparable. Our tests show that if the CALIOP-like solar filter were used in ATLID, one could lower the thresholds of Eq. (36) down to $\mathrm{SR}\left(\mathrm{532}\mathrm{nm},z\right)>\mathrm{2}$ and $\mathrm{ATB}\left(\mathrm{532}\mathrm{nm},z\right)-{\mathrm{ATB}}_{\mathrm{mol}}\left(\mathrm{532}\mathrm{nm},z\right)>\mathrm{1.0}\times {\mathrm{10}}^{-\mathrm{6}}{\mathrm{m}}^{-\mathrm{1}}{\mathrm{sr}}^{-\mathrm{1}}$ without losing the quality of cloud retrievals, whereas the same thresholds applied to CALIOP would give completely unacceptable results for daytime conditions.

Of course, the examples considered in this section do not cover the whole range of high-, middle-, and low-level clouds, but they draw a line between the threshold values that can be used for cloud definition for CALIOP and ATLID and show that the difference is linked to noise characteristics of the instruments. This result suggests that ATLID should be able to better observe optically thinner clouds than CALIOP in daytime at full horizontal resolution.

To illustrate this point, we used the available dataset for cirrus and estimated the minimal detectable backscatter (MDB) for ATLID in terms of equivalent ATB(532 nm,z) for comparison with CALIOP values obtained for 5 km horizontal averaging of cirrus measured at 15 km height (McGill et al., 2007). For this numerical experiment, we used noisy APBs and noise-free AMBs to keep the consistency with our approach of cloud detection using only one noisy component, the particulate one. For this horizontal averaging, we obtained MDB = 3.0 ± 1.0 × 10⁻⁷ m⁻¹ sr⁻¹ for the nighttime and MDB = 4.0 ± 1.0 × 10⁻⁷ m⁻¹ sr⁻¹ for the daytime in equivalent ATB(532 nm) values, whereas for CALIOP we obtained MDB = 4.0 ± 2.0 × 10⁻⁷ m⁻¹ sr⁻¹ for the nighttime and MDB = 1.3 ± 0.2 × 10⁻⁶ m⁻¹ sr⁻¹ for the daytime in its native ATB(532 nm). The daytime value estimated for CALIOP is in good agreement with the measured one (McGill et al., 2007), whereas the estimated nighttime value is somewhat lower than the measured MDB = 8.0 ± 1.0 × 10⁻⁷ m⁻¹ sr⁻¹. From this comparison, we cannot conclude that the ATLID will provide better sensitivity to thin clouds during nighttime, but we can conclude that its daytime thin-cloud detection at 5 km averaging capacity should be comparable to that of CALIOP for the nighttime, and this will be an important achievement for daytime vs. nighttime cloud comparison. Using the cloud detection thresholds defined by Eq. (36) and refined for the real data flow using the methodology outlined above, the CLIMP short-term product will be produced.

4.1 Capability of CLIMP and CALIPSO-GOCCP to detect the same clouds

One of the overarching goals of our study is to develop a method for merging the data from several spaceborne lidars into a continuous cloud record to detect long-term changes and get a seamless cloud climatology. Since the low threshold tested in the previous section revealed the sensitivity mismatch between the two instruments, we had to test whether the cloud detection thresholds developed for CALIOP (Chepfer et al., 2010) are applicable to ATLID and whether the clouds retrieved using these thresholds are consistent between the two lidars. For this exercise, we followed the same scheme as in the previous section, but this time the clouds were defined in Eq. (6) as in Chepfer et al. (2010) and the follow-up works (e.g., Cesana et al., 2019; Guzman et al., 2017).

Figure 10 demonstrates the daytime and nighttime scattering ratios above the detection thresholds (Eq. 6) and the corresponding cirrus cloud detection statistics for CALIOP. The SR(532 nm,z) in Fig. 10a and b demonstrates the same patterns as the ATB(532 nm,z) signals in Fig. 6a and b. As expected, this time the daytime noise is less pronounced (compare Fig. 10b to Fig. 8b). Still, the daytime scene contains a certain number of false detections marked in red in Fig. 10d. The same analysis performed for ATLID (Fig. 11) also shows somewhat less noise in daytime (compare Fig. 11b to Fig. 9b). The cloud detection quality of ATLID is like that of CALIOP (see Table 3). In this setup, the ATLID is just slightly better than CALIOP with its somewhat higher rate of false detections during the day (compare panels c and d of Figs. 8, 9, 10, and 11 and the corresponding columns in Table 3). For stratocumulus clouds (Figs. 1 and 5), with their strong signals, the agreement between CALIOP and ATLID is also better than for the clouds defined by Eq. (36) (compare panels c and d of Figs. A1, A2, A3, and A4). The 1 km averaging further improves the agreement between the datasets (Table 3).

Figure 10Scattering ratio and cloud detection statistics estimated for cirrus clouds observed by CALIOP using Eq. (6): (a) scattering ratio at night, (b) cloud detection at night, (c) scattering ratio during the day, and (d) cloud detection during the day. Note the color scale difference for (a) and (c) compared to (b) and (d).

Download

Summarizing, using the thresholds (Eq. 6) to define the clouds makes the cloud datasets from CALIOP and ATLID comparable. Further adjustment will be needed for real ATLID data to compensate the effects of the diurnal cycle (Noel et al., 2018; Chepfer et al., 2019; Feofilov and Stubenrauch, 2019). Other compensations might be required when the real ATLID data become available. Since there is a high chance that there will be no overlapping period for these two satellite instruments, an intercalibration procedure will be required. For this, one can use the average cloud amount for low, middle, and high clouds in different zones (tropics, middle latitudes, and polar) to track the changes and to introduce feedback to the cloud detection algorithm. This way, the number of cases measured for each zone will be high, and the uncertainty will be low. The daytime and nighttime observations should be considered separately to address the diurnal cycle and daytime noise issues. In the sections below, we assume that the intercalibration has been performed and that the cloud datasets agree.

Table 3Cloud detection statistics for CALIOP and ATLID in the case when the cloud is defined as $\mathrm{SR}\left(\mathrm{532}\mathrm{nm},z\right)>\mathrm{5}$ and $\mathrm{ATB}\left(\mathit{\lambda },z\right)-{\mathrm{ATB}}_{\mathrm{mol}}\left(\mathit{\lambda },z\right)>\mathrm{2.5}\times {\mathrm{10}}^{-\mathrm{6}}{\mathrm{m}}^{-\mathrm{1}}{\mathrm{sr}}^{-\mathrm{1}}$ (Eq. 6).

Download Print Version | Download XLSX

4.2 Numerical chain to simulate long-term lidar record and method to quantify time of emergence

The previous section shows that ATLID and CALIOP data may be merged to build a long-term dataset, even though their instrumental or orbital differences might necessitate further reconciliation. Here we suppose perfect reconciliation will eventually be reached, and we build a long-term space lidar synthetic dataset spanning more than 30 years to examine when a change in cloud properties attributable to human-induced warming would be detectable in the lidar cloud record according to climate model simulations. This approach is directly inspired by the one pioneered in Chepfer et al. (2018) and later expanded in Perpina et al. (2021).

We use climate predictions from IPSL-CM6 (Institut Pierre Simon Laplace climate model; Boucher et al., 2020) and CESM2 (Community Earth System Model; Hurrell et al., 2013), two ocean–atmosphere coupled GCMs which took part in Climate Model Intercomparison Project (CMIP) Phase 6 (Eyring et al., 2016). We use predictions that start in 2008 and end in 2034 and follow the RCP8.5 (Representative Concentration Pathway) scenario, which tracks the observed CO₂ emissions closely (Schwalm et al., 2020). Predictions are provided as monthly grids with spatial resolutions of 1.27^∘ × 2.5^∘ on 79 vertical levels (IPSL-CM6) and 1.25^∘ × 0.94^∘ on 40 vertical levels (CESM). On these predictions of atmospheric conditions, we apply the COSP1.4 lidar simulator (Sect. 3.2), which generates on similar spatial grids the monthly averaged cloud properties that would be observed by a spaceborne lidar flying over the simulated atmosphere. In addition to the simulation steps described in Sect. 3.2, here, as a first step of the simulation, for each grid box of the GCM-created atmosphere an ensemble of subgrid-scale profiles are stochastically generated by the Subgrid Cloud Overlap Profile Sampler (Klein and Jakob, 1999). Each of these profiles is fed to the COSP simulator, which generates a synthetic lidar profile, on which cloud detection is performed. All subgrid-scale cloud detection profiles are eventually averaged to generate a single vertical profile for each grid box (see Chepfer et al., 2008, for details).

From the synthetic cloud properties, we considered two climate diagnostics whose trend should be related to climate change: first the fraction of opaque clouds C_opaque, defined as the number of lidar profiles in which an opaque cloud is detected in a given latitude–longitude grid box divided by the total number of profiles sampled in the same grid box. Opaque clouds are responsible for the majority of cloud radiative effects in the tropics (Vaillant de Guélis et al., 2017), and the cloud amount has been identified as one of the main drivers of cloud feedbacks on climate (Zelinka et al., 2016); thus the fraction of opaque clouds should be closely tied to climate change. Second, we considered the altitude of full attenuation Z_opaque (Guzman et al., 2017) averaged over all opaque profiles in every grid box. The vertical distribution of clouds is closely linked to their longwave radiative impact and to climate change (Vaillant de Guelis et al., 2018), and their altitude is expected to increase by several hundred meters per century (Richardson et al., 2022). Altitude is among the cloud properties whose change is expected to be detectable the earliest using active remote sensing (Chepfer et al., 2014; Takahashi et al., 2019; Aerenson et al., 2022).

From the GCM predictions, the COSP lidar simulator generates monthly grids of C_opaque and Z_opaque that we spatially average over the tropics (30^∘ S–30^∘ N) to get monthly time series. We deseasonalize those time series to get their monthly anomalies over the 2008–2034 period. For any time t along these time series, the record length is equivalent to the period between 1 January 2008 and t, and we computed the trend w(t) as the linear regression of the time series of anomalies over that period. The uncertainty σ_w(t) in the trend w(t) at time t was computed, as in Chepfer et al. (2018), as ${\mathit{\sigma }}_{\mathrm{w}}\left(t\right)={\mathit{\sigma }}_{\mathrm{N}}\sqrt{\frac{\mathrm{1}+\mathit{\phi }}{\mathrm{1}-\mathit{\phi }}}{n}^{-\frac{\mathrm{3}}{\mathrm{2}}}$, with n being the number of years in the record at time t, φ being the lag−1 autocorrelation coefficient of the series between 0 and t, and σ_N being the standard deviation of the noise remaining in the series between 0 and t once it has been deseasonalized and the autocorrelated part has been removed.

The following analysis focuses on the tropical regions (30^∘ S–30^∘ N), where the atmospheric circulation will be impacted by the weakening of the Hadley and Walker circulations expected in the upcoming century by most climate predictions (Davis and Rosenlof, 2012; Su et al., 2014; Kjellsson, 2015; Chemke, 2021). These changes will have important effects on the spatial distribution of tropical clouds (Su et al., 2014), which provide the basis for our climate diagnostics. Cloud opacity is one of the cloud properties most closely linked to their radiative impact (Zelinka et al., 2012), which explains why our diagnostics are based on the properties of opaque clouds (as in Perpina et al., 2021). The results below assume it will be possible to process ATLID measurements in such a way that CLIMP and GOCCP cloud properties are consistent.

4.3 How many years of ATLID observation are required in addition to CALIPSO to evaluate the climate model prediction of cloud changes?

Figure 12 shows how the uncertainty in the retrieved trend for C_opaque changes with the length of the record of lidar-based cloud properties, starting in 2008, according to predictions from IPSL-CM6 (blue) and CESM1 (orange). The uncertainty is generally the largest and fluctuates most when the record is short and decreases and stabilizes as the record gets longer. At any time t if we require a 95 % confidence level in the prediction and assume trends are normally distributed, the real trend will lie in the w(t)±2σ_w(t) interval. The sign of the trend will be robust once $\left|\frac{w\left(t\right)}{{\mathit{\sigma }}_{\mathrm{w}}\left(t\right)}\right|>\mathrm{2}$. This is when the uncertainty in the trend becomes small compared to the trend itself and marks the time of emergence of cloud change induced by anthropogenic warming. This occurs earlier for strong, stable trends and might never occur for very small trends or trends whose sign changes over time. Times of emergence in the C_opaque time series are indicated in Fig. 12 with triangles for three confidence levels (50 %, 70 %, and 95 %). Reaching a reliable sign requires a longer record if the required confidence level is strong.

Figure 11Same as Fig. 10 but for ATLID.

Download

Figure 12Evolution of the uncertainty in the C_opaque trend as a function of the length of the spaceborne lidar record, according to atmospheric conditions predicted by IPSL-CM6 (blue; Laboratoire de Météorologie Dynamique, LMD) and CESM (orange) in the period between 2008 and 2034 following the RCP8.5 scenario. The first 2 years of the record (2008–2010) are considered in the analysis, but trend uncertainties during that period are very large and are masked in the figure to improve the legibility of later years. CALIPSO's (CAL) planned end of operation (2023) is marked by a vertical blue line. Supposing EarthCARE (EC) begins operation right afterward, its nominal 3-year operation point is marked by a vertical purple line, and an optimistic 10-year operation point is marked by a vertical black line.

Download

According to predictions from IPSL-CM6 (blue), a reliable trend should emerge from the natural variability at a 50 % to 70 % confidence level between 2030 and 2032. In other words, IPSL-CM6 predicts that revealing a reliable long-term trend in the fraction of opaque clouds would require an uninterrupted spaceborne lidar record of 22 years, which would be achievable if EarthCARE operates for at least 7 years. Reaching 95 % confidence levels on the retrieved trend would require extending the record beyond 25 years, most probably through another spaceborne lidar mission further in time. CESM1, meanwhile, predicts that a reliable long-term trend in the fraction of opaque clouds (at similar confidence levels between 50 % and 70 %) would be reached between 2025 and 2027, requiring 2 to 4 years of EarthCARE operation. A highly reliable trend (95 % confidence levels) would be detectable in 2029, after 6 years of EarthCARE operation. In summary, if a 50 % confidence level is acceptable, detecting a reliable trend would either be possible within the EarthCARE nominal operation time frame (2 years after launch), according to CESM1, or require EarthCARE to operate 4 years beyond its planned lifetime, according to IPSL-CM6.

If we consider the Z_opaque diagnostic (Fig. 13), the IPSL-CM6 model now predicts a trend will be detectable at high 95 % confidence levels in 2024, i.e., 1 year into EarthCARE's nominal operation period. Meanwhile, according to CESM1 predictions, detecting a reliable trend (even at a modest 50 % confidence level) would require EarthCARE operating for 8 years, 5 years beyond its nominal operation time frame. This very fast detection of a reliable Z_opaque trend predicted by IPSL-CM6 is consistent with how this model expects important and fast changes in the vertical distribution of opaque clouds in the tropics (Perpina et al., 2021).

Figure 13Same as Fig. 12 but for the altitude of opacity Z_opaque instead of C_opaque.

Download

We sum these results up in Table 4, which in addition provides similar record lengths to detect reliable trends when considering grid boxes dominated by either low or high clouds. Tropical low opaque clouds include sparse shallow cumulus (Konsta et al., 2012) and optically thicker stratocumulus along the western coasts of continents (Guzman et al., 2017), both confined to the boundary layer and most frequent in subsidence regions. By contrast, tropical high opaque clouds are more localized and strongly correlated with deep convection. Since both kinds of clouds are driven by very different processes, it is not unreasonable to assume they will probably evolve differently in the upcoming century, which justifies their separate studies. In practice, we identified grid boxes dominated by low clouds as those where Z_opaque was below 3 km and high-cloud grid boxes as those where Z_opaque was above 3 km. The results, shown in Table 4, suggest that the nominal ATLID (EarthCARE) operation will be enough to validate or invalidate the trends in opaque tropical low clouds predicted by CESM. It will be possible to validate or invalidate other model-based cloud predictions only if EarthCARE performs beyond its nominal lifetime (which is not impossible, as CALIPSO demonstrated) or if measurements from a follow-up spaceborne lidar mission after ATLID are included in the cloud profile record. These results are consistent with the trends, uncertainties, and times of emergence found when conducting a relatively simpler comparison of HadGEM2-A (Hadley Centre Global Environment Model) predictions in current vs. +4 K conditions (Chepfer et al., 2018). Needless to say that the treatment of any follow-up mission (e.g., Atmosphere Observing System, AOS; Aeolus-2) will require the compensation for all the differences between the lidars like it is done in this work.

Table 4When will a spaceborne lidar record starting in 2008 be long enough to enable a reliable detection (at 70 % confidence level) of C_opaque or Z_opaque trends according to predictions from IPSL-CM6 or CESM? The required years of EarthCARE operation are shown in parentheses, supposing they begin in 2023. The monthly evolution of the trend uncertainties for low and high clouds are provided in Figs. B1 and B2 of Appendix B.

Download Print Version | Download XLSX

As stated up front, these results depend on rather strong hypotheses of perfect continuity and perfect intercalibration between the consecutive spaceborne lidars that provide the measurements from which the cloud properties are derived. Imperfect continuity would occur if, for instance, EarthCARE starts operation later than CALIOP stops. The missing years in the record would delay the detection of a reliable trend by at least the same time period (Chepfer et al., 2018). Perfect intercalibration supposes the effects of instrumental differences in technical specifications (wavelengths, pulse energy, field of view, etc.) and orbital characteristics (local time of overpass, altitude) on lidar measurements are reconciled somehow. For instance, ATLID operates at 355 nm and CALIOP operates at 532 nm, and the impact this has on measurements can be reconciled by converting the ATLID signal at 532 nm as done in the current study, but the costs of this conversion are not completely understood and will require re-examination when actual ATLID data are available. Imperfect intercalibration could lead to offsets in one spaceborne lidar's record compared to the other and would increase the uncertainties in the retrieved trends. Increased delays between the operation of both instruments would complicate their intercalibration. The different local times of overpass (01:30 and 13:30 local solar time, LST, for CALIPSO; 06:00 and 18:00 LST for EarthCARE) are also quite problematic, since each instrument will sample clouds at a different phase of their diurnal cycle (Noel et al., 2018; Chepfer et al., 2019; Feofilov and Stubenrauch, 2019). In particular, this will impact high clouds related to deep convection that exhibit a marked diurnal cycle. It is out of the scope of the present work to evaluate how this change could bias the retrieved long-term trends. The same applies to a follow-up lidar mission, which may or may not operate at the same orbit with the same local solar time of overpass and may or may not measure the depolarized backscatter.

Finally, the times of emergence presented here must not be understood as definite but as predictions by climate models. It is worth noting, for instance, that, according to predictions from IPSL-CM6, a reliable trend should already be readily detectable in the existing record of Z_opaque that is today only built on CALIOP (CALIPSO) (Table 4). Such a trend has not been identified yet. This is consistent with the fact that in current climate conditions IPSL-CM6 overestimates the altitude of opaque clouds in tropical convective regions and brings them significantly higher (+ 2 km) near the end of 21st century (Perpina et al., 2021). Such rapid changes are not present in CESM predictions. These important model differences highlight the crucial need for continued long-term cloud lidar observations able to monitor the actual cloud changes and disambiguate model predictions.