Self-calibration dense bundle adjustment of multi-view Worldview-3 basic images
Introduction
Since the launch of Worldview-3 and Worldview-4, the ground sample distance (GSD) of very high-resolution satellite images (VHRSIs) has reached a remarkable 0.31 m. The new satellite constellation of AIRBUS, Pléiades-Neo, offers an alternative VHRSI of 30 cm. These images provide large-scale detailed urban information, such as individual trees, cars, and architectural structures (Loghin et al., 2020), which were only available in aerial images in the past. Geometric and semantic information can be extracted from multi-view VHRSIs, which is a common interest of both photogrammetry and remote sensing and the computer vision society (Bosch et al., 2019, d'Angelo and Kuschk, 2012, Kunwar et al., 2020, Lefèvre et al., 2017). In 2016, IAPRA launched the “Multi-View Stereo 3D Challenge” (Bosch et al., 2016), which provided a large volume of data to the public to promote the development of algorithms and software. The automatic 3D reconstruction of VHRSIs is widely used to generate a digital surface model (DSM) (Melet et al., 2020, Perko et al., 2019). Many researchers attempted to select stereo pairs (Facciolo et al., 2017, Qin, 2019) and developed dense image-matching methods for VHRSIs (Noh and Howat, 2017, Rothermel et al., 2020, Rupnik et al., 2018). Although the winner of IAPRA achieved the best result with selected stereo pairs (Facciolo et al., 2017), we suggest that redundant observations with consistency would improve the results, which is guaranteed via bundle adjustment. However, the remarkable bundle adjustment of VRHRIs suffers from a local bundle adjustment (Facciolo et al., 2017), inconsistent stereo pairs (Marí et al., 2019), and large reprojection errors (Zhang et al., 2019).
To retrieve imaging rays, bundle adjustment with subpixel accuracy is a critical step. After bundle adjustment, all corresponding rays intersect with each other, thus establishing the epipolar geometry and consistency of the stereo pairs. For 3D reconstruction, it is important to reduce the dense matching search space from two dimensions to one and to set the same datum for multi-stereo pair fusion. Bias compensation (Fraser and Hanley, 2003), or bundle adjustment with a bias compensation model (Grodecki and Dial, 2003), has achieved enormous VHRSIs improvements since the achievements of IKONOS. These methods are based on the hypothesis that VHRSIs are carefully processed, and only errors in the exterior orientation parameters (EOPs) need to be compensated. Bundle adjustment of VHRSIs needs to be carefully performed if there are errors in the interior orientation parameters (IOPs) and attitude jitters.
Indeed, attitude jitter or oscillation is detrimental to VHRSIs. Under a small GSD and large distance, an extremely small amount of jitter introduces pixel-level distortions. Attitude jitter has been determined in both earth (Ayoub et al., 2008, Lehner and Müller, 2003, Teshima and Iwasaki, 2008, Ye et al., 2019) and planet (Kirk et al., 2008) explosions. To detect the attitude jitter of VHRSIs, Jacobsen (2018) adopted the SRTM or AW3D30 DSM as a reference DSM to evaluate the subpixel y-parallax after bias compensation with the rational function model (RFM). However, a flaw in the RFM, i.e., its inability to handle an attitude jitter owing to its smoothness, has not received sufficient attention. In this study, we verified that the basic images of Worldview-3 require a self-calibrated bundle adjustment of the rigorous sensor model (RSM) to compensate for attitude jitter and IOP errors. We used a dense bundle adjustment instead of a sparse bundle adjustment for a multi-view VHRSI orientation to comprehensively explore geometric properties. Our experimental results contradicted previously drawn conclusions that the RFM could achieve an accuracy comparable to that of an RSM (Hong et al., 2015, Robertson, 2003, Teo, 2011).
The remainder of this paper is organized as follows. Section 2 briefly introduces related studies. In Section 3, the attitude-jitter-compatible RSM for the basic product of Worldview-3 is presented, followed by attitude-jitter filtering of the RFM. A self-calibration dense bundle adjustment method is proposed in Section 4. In Section 5, we describe the experiments conducted. Finally, Section 6 summarizes the conclusions of the study.
Section snippets
Related works
The bundle adjustment of VHRSIs depends on the geometric sensor model. The RSM retrieves the imaging ray from the image space to the ground coordinate system (Poli and Toutin, 2012, Toutin, 2004). In general, there are two different RSM strategies for addressing the following attitudes: bundle adjustment with a simple polynomial attitude model and bundle adjustment with a bias compensation model. The former assumes that the attitude of VHRSIs is steady, which can be modeled using polynomial
Modeling attitude jitter using RSM
The basic imagery products of Worldview-3 were radiometrically corrected and sensor-corrected. To stitch multiple charge‐coupled device detectors (CCDs), a synthetic array was used to generate a single image. Synthetic camera information, attitude, and ephemeris were supplied to users to build the RSM, and RFM is also provided.
Dense bundle adjustment
The bundle adjustment of VHRSIs minimizes the weighted square of residuals in the image space. The distribution and number of observations, including the GCPs and TPs, are crucial. The sparse bundle adjustment utilizes a limited number of sparsely distributed TPs and GCPs to estimate a few adjustment parameters. However, the attitude jitter introduces high-frequency distortions into the images. The bundle adjustment with sparse TPs may miss the attitude jitter. To detect the attitude jitter of
Datasets
The Applied Physical Laboratory of Johns Hopkins University provided the first large benchmark dataset for multi-view stereo satellite images to the public, which included 47 basic imagery scenes from Worldview-3 and a high-resolution airborne Lidar with a range of approximately 20 km2. Only panchromatic scenes were used in the experiments. These data were collected between November 2014 and January 2016, covering San Fernando, Argentina. Most images were reprocessed on December 2, 2015. The
Conclusions
Bundle adjustment of VHRSIs has been studied over the past 20 years since the launch of IKONOS. With a sparse bundle adjustment, existing studies concluded that RFM could achieve an accuracy comparable to that of the RSM. In this study, we highlighted that RSM can compensate attitude jitters for VHRSIs, whereas RFM introduces significant errors because it filters the attitude jitters. Self-calibration of a dense bundle adjustment, which applied RSM for the basic product imagery of Worldview-3,
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
The authors would like to appreciate the Applied Physics Laboratory of Johns Hopkins University and Digital Globe for providing the experimental datasets. This study was supported by the National Natural Science Foundation of China (NSFC) project (Nos. 41971418 and 41701538) and the Gaofen Satellite Remote Sensing Surveying and Mapping Application Demonstration System (Phase II) of China (No. 42-Y30B04-9001-19/21).
References (69)
- et al.
Nonlinear bias compensation of ZiYuan-3 satellite imagery with cubic splines
ISPRS J. Photogram. Remote Sens.
(2017) Rigorous photogrammetric processing of SPOT images at CCM Canada
ISPRS J. Photogramm. Remote Sens.
(1989)Atmospheric refraction effects in Earth remote sensing
ISPRS J. Photogramm. Remote Sens.
(1999)- et al.
The Surface Extraction from TIN based Search-space Minimization (SETSM) algorithm
ISPRS J. Photogramm. Remote Sens.
(2017) - et al.
Precise georeferencing using the rigorous sensor model and rational function model for ZiYuan-3 strip scenes with minimum control
ISPRS J. Photogramm. Remote Sens.
(2016) A critical analysis of satellite stereo pairs for digital surface model generation and a matching quality prediction model
ISPRS J. Photogramm. Remote Sens.
(2019)- et al.
Photometric multi-view mesh refinement for high-resolution satellite images
ISPRS J. Photogramm. Remote Sens.
(2020) - et al.
3D reconstruction from multi-view VHR-satellite images in MicMac
ISPRS J. Photogramm. Remote Sens.
(2018) - et al.
Correcting bias in the rational polynomial coefficients of satellite imagery using thin-plate smoothing splines
ISPRS J. Photogramm. Remote Sens.
(2017) - et al.
Detection and estimation of ZY-3 three-line array image distortions caused by attitude oscillation
ISPRS J. Photogramm. Remote Sens.
(2015)
Bias-corrected rational polynomial coefficients for high accuracy geo-positioning of QuickBird stereo imagery
ISPRS J. Photogramm. Remote Sens.
Correction of ZY-3 image distortion caused by satellite jitter via virtual steady reimaging using attitude data
ISPRS J. Photogramm. Remote Sens.
Correction of atmospheric refraction geolocation error for high resolution optical satellite pushbroom images
Photogramm. Eng. Remote Sens.
Estimation and analysis of along-track attitude jitter of ZiYuan-3 satellite based on relative residuals of tri-band multispectral imagery
ISPRS J. Photogramm. Remote Sens.
Quality assessment of digital surface models extracted from WorldView-2 and WorldView-3 stereo pairs over different land covers
GISci. Remote Sens.
Georeferencing accuracy analysis of a single worldview-3 image collected over Milan
Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.
WorldView-3 geometric calibration
Dense multi-view stereo from satellite imagery
An evaluation of rational functions for photogrammetric restitution
Int. Arch. Photogram. Remote Sens.
Bias compensation in rational functions for IKONOS satellite imagery
Photogramm. Eng. Rem. S.
Bias-compensated RPCs for sensor orientation of high-resolution satellite imagery
Photogramm. Eng. Rem. S.
Block adjustment of high-resolution satellite images described by rational polynomials
Photogramm. Eng. Rem. S.
Methods for computing photogrammetric refraction corrections for vertical and oblique photographs
Photogramm. Eng. Remote Sens.
A comparison of the performance of bias-corrected RSMs and RFMs for the geo-positioning of high-resolution satellite stereo imagery
Remote Sens.
DEM extraction from Worldview-3 stereo-images and accuracy evaluation
Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.
Research on Compensation for System Errors of Basic Satellite Products, State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS)
Cited by (8)
Accuracy Assessment of GF-7 Geolocation Without GCPs Considering Atmospheric Refraction and Aberration of Light
2024, IEEE Transactions on Geoscience and Remote SensingModel-Driven Deep Pipeline With Uncertainty-Aware Bundle Adjustment for Satellite Photogrammetry
2024, IEEE Transactions on Geoscience and Remote SensingLarge-Scale Block Bundle Adjustment of LROC NAC Images for Lunar South Pole Mapping Based on Topographic Constraint
2024, IEEE Journal of Selected Topics in Applied Earth Observations and Remote SensingRigorous Sensor Model of Gaofen-7 Satellite Laser Altimeter Based on Coupled Footprint Camera
2023, IEEE Journal of Selected Topics in Applied Earth Observations and Remote SensingPre-Processing of Inner CCD Image Stitching of the SDGSAT-1 Satellite
2022, Applied Sciences (Switzerland)