Self-calibration dense bundle adjustment of multi-view Worldview-3 basic images

https://doi.org/10.1016/j.isprsjprs.2021.04.013Get rights and content

Abstract

Very high-resolution satellite images (VHRSIs) with improved spatial resolution provide unprecedented opportunities to explore the geometric and semantic information of the world. Accordingly, to compensate for the bias of exterior orientation parameters, bundle adjustment of VHRSIs is required. Owing to the sparse points, the bundle adjustment of a few VHRSIs omits high-frequency errors and attitude jitters; therefore, the rational function model (RFM) can achieve an accuracy comparable to that of a rigorous sensor model (RSM), despite the significant difference between the RSM and RFM. In this study, we provide insight into the role of RSM in modeling attitude jitters for the Worldview-3 basic imagery product. Penalized splines were proposed to model the attitude jitters. After correcting the photogrammetric refraction using Saastamoinen’s model and light aberration, the RSM was built for agile satellites. The difference between RSM and RFM is consistent with attitude jitters, which is calculated using the penalized splines model and third-degree polyniomials. To fully explore the attitude jitters, a dense bundle adjustment was proposed to process 47 scenes of the Worldview-3 basic product imagery, which was provided by Johns Hopkins University Applied Physics Laboratory for the “Multi-View Stereo 3D Challenge.” Pairwise feature matching and feature tracking were adopted to generate over 16 000 tie-points (TPs), which were detected in an average of 9.38 images. The bundle adjustment residuals with RFM exhibited distortions similar to those of the attitude jitter. The experiments verified that the bundle adjustment with RFM introduced significant errors triggered by attitude jitters, with a maximum of over 5.0 pixels. The bundle adjustment with RSM could eliminate significant errors caused by attitude jitters and reduce the root mean square errors (RMSEs) from 1.12 pixels to 0.61 pixels. However, the basic product imagery of Worldview-3 exhibited errors in the interior orientation parameters. After analyzing the physical meanings of bias compensation, a self-calibration model was proposed. After comparing the shift compensation, affine compensation, self-calibration, temporal self-calibration, second-degree self-calibration, and second-degree polynomial models, the dense bundle adjustment with self-calibration was suggested because it could compensate for errors in the interior orientation parameters (IOPs) and obtain an accuracy similar to that of high-degree models. Thus, a self-calibration dense bundle adjustment with RSM compensates for the attitude jitters and errors in the IOPs, and achieves a remarkable accuracy of 0.49 pixels in the image coordinates.

Introduction

Since the launch of Worldview-3 and Worldview-4, the ground sample distance (GSD) of very high-resolution satellite images (VHRSIs) has reached a remarkable 0.31 m. The new satellite constellation of AIRBUS, Pléiades-Neo, offers an alternative VHRSI of 30 cm. These images provide large-scale detailed urban information, such as individual trees, cars, and architectural structures (Loghin et al., 2020), which were only available in aerial images in the past. Geometric and semantic information can be extracted from multi-view VHRSIs, which is a common interest of both photogrammetry and remote sensing and the computer vision society (Bosch et al., 2019, d'Angelo and Kuschk, 2012, Kunwar et al., 2020, Lefèvre et al., 2017). In 2016, IAPRA launched the “Multi-View Stereo 3D Challenge” (Bosch et al., 2016), which provided a large volume of data to the public to promote the development of algorithms and software. The automatic 3D reconstruction of VHRSIs is widely used to generate a digital surface model (DSM) (Melet et al., 2020, Perko et al., 2019). Many researchers attempted to select stereo pairs (Facciolo et al., 2017, Qin, 2019) and developed dense image-matching methods for VHRSIs (Noh and Howat, 2017, Rothermel et al., 2020, Rupnik et al., 2018). Although the winner of IAPRA achieved the best result with selected stereo pairs (Facciolo et al., 2017), we suggest that redundant observations with consistency would improve the results, which is guaranteed via bundle adjustment. However, the remarkable bundle adjustment of VRHRIs suffers from a local bundle adjustment (Facciolo et al., 2017), inconsistent stereo pairs (Marí et al., 2019), and large reprojection errors (Zhang et al., 2019).

To retrieve imaging rays, bundle adjustment with subpixel accuracy is a critical step. After bundle adjustment, all corresponding rays intersect with each other, thus establishing the epipolar geometry and consistency of the stereo pairs. For 3D reconstruction, it is important to reduce the dense matching search space from two dimensions to one and to set the same datum for multi-stereo pair fusion. Bias compensation (Fraser and Hanley, 2003), or bundle adjustment with a bias compensation model (Grodecki and Dial, 2003), has achieved enormous VHRSIs improvements since the achievements of IKONOS. These methods are based on the hypothesis that VHRSIs are carefully processed, and only errors in the exterior orientation parameters (EOPs) need to be compensated. Bundle adjustment of VHRSIs needs to be carefully performed if there are errors in the interior orientation parameters (IOPs) and attitude jitters.

Indeed, attitude jitter or oscillation is detrimental to VHRSIs. Under a small GSD and large distance, an extremely small amount of jitter introduces pixel-level distortions. Attitude jitter has been determined in both earth (Ayoub et al., 2008, Lehner and Müller, 2003, Teshima and Iwasaki, 2008, Ye et al., 2019) and planet (Kirk et al., 2008) explosions. To detect the attitude jitter of VHRSIs, Jacobsen (2018) adopted the SRTM or AW3D30 DSM as a reference DSM to evaluate the subpixel y-parallax after bias compensation with the rational function model (RFM). However, a flaw in the RFM, i.e., its inability to handle an attitude jitter owing to its smoothness, has not received sufficient attention. In this study, we verified that the basic images of Worldview-3 require a self-calibrated bundle adjustment of the rigorous sensor model (RSM) to compensate for attitude jitter and IOP errors. We used a dense bundle adjustment instead of a sparse bundle adjustment for a multi-view VHRSI orientation to comprehensively explore geometric properties. Our experimental results contradicted previously drawn conclusions that the RFM could achieve an accuracy comparable to that of an RSM (Hong et al., 2015, Robertson, 2003, Teo, 2011).

The remainder of this paper is organized as follows. Section 2 briefly introduces related studies. In Section 3, the attitude-jitter-compatible RSM for the basic product of Worldview-3 is presented, followed by attitude-jitter filtering of the RFM. A self-calibration dense bundle adjustment method is proposed in Section 4. In Section 5, we describe the experiments conducted. Finally, Section 6 summarizes the conclusions of the study.

Section snippets

Related works

The bundle adjustment of VHRSIs depends on the geometric sensor model. The RSM retrieves the imaging ray from the image space to the ground coordinate system (Poli and Toutin, 2012, Toutin, 2004). In general, there are two different RSM strategies for addressing the following attitudes: bundle adjustment with a simple polynomial attitude model and bundle adjustment with a bias compensation model. The former assumes that the attitude of VHRSIs is steady, which can be modeled using polynomial

Modeling attitude jitter using RSM

The basic imagery products of Worldview-3 were radiometrically corrected and sensor-corrected. To stitch multiple charge‐coupled device detectors (CCDs), a synthetic array was used to generate a single image. Synthetic camera information, attitude, and ephemeris were supplied to users to build the RSM, and RFM is also provided.

Dense bundle adjustment

The bundle adjustment of VHRSIs minimizes the weighted square of residuals in the image space. The distribution and number of observations, including the GCPs and TPs, are crucial. The sparse bundle adjustment utilizes a limited number of sparsely distributed TPs and GCPs to estimate a few adjustment parameters. However, the attitude jitter introduces high-frequency distortions into the images. The bundle adjustment with sparse TPs may miss the attitude jitter. To detect the attitude jitter of

Datasets

The Applied Physical Laboratory of Johns Hopkins University provided the first large benchmark dataset for multi-view stereo satellite images to the public, which included 47 basic imagery scenes from Worldview-3 and a high-resolution airborne Lidar with a range of approximately 20 km2. Only panchromatic scenes were used in the experiments. These data were collected between November 2014 and January 2016, covering San Fernando, Argentina. Most images were reprocessed on December 2, 2015. The

Conclusions

Bundle adjustment of VHRSIs has been studied over the past 20 years since the launch of IKONOS. With a sparse bundle adjustment, existing studies concluded that RFM could achieve an accuracy comparable to that of the RSM. In this study, we highlighted that RSM can compensate attitude jitters for VHRSIs, whereas RFM introduces significant errors because it filters the attitude jitters. Self-calibration of a dense bundle adjustment, which applied RSM for the basic product imagery of Worldview-3,

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

The authors would like to appreciate the Applied Physics Laboratory of Johns Hopkins University and Digital Globe for providing the experimental datasets. This study was supported by the National Natural Science Foundation of China (NSFC) project (Nos. 41971418 and 41701538) and the Gaofen Satellite Remote Sensing Surveying and Mapping Application Demonstration System (Phase II) of China (No. 42-Y30B04-9001-19/21).

References (69)

  • X. Tong et al.

    Bias-corrected rational polynomial coefficients for high accuracy geo-positioning of QuickBird stereo imagery

    ISPRS J. Photogramm. Remote Sens.

    (2010)
  • M. Wang et al.

    Correction of ZY-3 image distortion caused by satellite jitter via virtual steady reimaging using attitude data

    ISPRS J. Photogramm. Remote Sens.

    (2016)
  • M. Yan et al.

    Correction of atmospheric refraction geolocation error for high resolution optical satellite pushbroom images

    Photogramm. Eng. Remote Sens.

    (2016)
  • Z. Ye et al.

    Estimation and analysis of along-track attitude jitter of ZiYuan-3 satellite based on relative residuals of tri-band multispectral imagery

    ISPRS J. Photogramm. Remote Sens.

    (2019)
  • M.A. Aguilar et al.

    Quality assessment of digital surface models extracted from WorldView-2 and WorldView-3 stereo pairs over different land covers

    GISci. Remote Sens.

    (2019)
  • ASTRIUM, 2012. Pléiades Imagery User...
  • Ayoub, F., Leprince, S., Binet, R., Lewis, K.W., Aharonson, O., Avouac, J.P., 2008. Influence of camera distortions on...
  • L. Barazzetti et al.

    Georeferencing accuracy analysis of a single worldview-3 image collected over Milan

    Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

    (2016)
  • Bosch, M., Foster, K., Christie, G., Wang, S., Hager, G.D., Brown, M., 2019. Semantic stereo for incidental satellite...
  • Bosch, M., Kurtz, Z., Hagstrom, S., Brown, M., 2016. A multiple view stereo benchmark for satellite imagery. In:...
  • Bresnahan, P., Brown, E., HenryVazquez, L., 2015. WorldView-3 Absolute Geolocation Accuracy Evaluation. In: Joint...
  • C. Comp et al.

    WorldView-3 geometric calibration

    (2015)
  • P. d'Angelo et al.

    Dense multi-view stereo from satellite imagery

  • DigitalGlobe, 2012. DigitalGlobe Core Imagery Products...
  • I. Dowman et al.

    An evaluation of rational functions for photogrammetric restitution

    Int. Arch. Photogram. Remote Sens.

    (2000)
  • Facciolo, G., Franchis, C.D., Meinhardt-Llopis, E., 2017. Automatic 3D Reconstruction from Multi-date Satellite Images....
  • C.S. Fraser et al.

    Bias compensation in rational functions for IKONOS satellite imagery

    Photogramm. Eng. Rem. S.

    (2003)
  • C.S. Fraser et al.

    Bias-compensated RPCs for sensor orientation of high-resolution satellite imagery

    Photogramm. Eng. Rem. S.

    (2005)
  • J. Grodecki et al.

    Block adjustment of high-resolution satellite images described by rational polynomials

    Photogramm. Eng. Rem. S.

    (2003)
  • M.S. Gyer

    Methods for computing photogrammetric refraction corrections for vertical and oblique photographs

    Photogramm. Eng. Remote Sens.

    (1996)
  • Z.H. Hong et al.

    A comparison of the performance of bias-corrected RSMs and RFMs for the geo-positioning of high-resolution satellite stereo imagery

    Remote Sens.

    (2015)
  • F. Hu et al.

    DEM extraction from Worldview-3 stereo-images and accuracy evaluation

    Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

    (2016)
  • W. Huang

    Research on Compensation for System Errors of Basic Satellite Products, State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS)

    (2016)
  • Jacobsen, K., 2017. Problems and limitations of satellite image orientation for determination of height models. In: The...
  • View full text