Land-cover classification with high-resolution remote sensing images using transferable deep models
Introduction
Land-cover classification with remote sensing (RS) images plays an important role in many applications such as land resource management, urban planning, precision agriculture, and environmental protection (Mathieu et al., 2007; Shi et al., 2015; Ozdarici-Ok et al., 2015; Zhang and Kovacs, 2012; Ardila et al., 2011; Fauvel et al., 2013). In recent years, high-resolution remote sensing (HRRS) images are increasingly available. Meanwhile, multi-source and multi-temporal RS images can be obtained over different geographical areas (Moser et al., 2013). Such large amount of heterogeneous HRRS images provide detailed information of the land surface, and therefore open new avenues for large-coverage and multi-temporal land-cover mapping. However, the rich details of objects emerging in HRRS images, such as the geometrical shape and structural content of objects, bring more challenges to land-cover classification (Bruzzone and Carlin, 2006). Furthermore, diverse imaging conditions usually lead to photographic distortions, variations in scale and changes of illumination in RS images, which often seriously reduces the separability among different classes (Tuia et al., 2016). Due to these influences, optimal classification models learned from certain annotated images always quickly lose their effectiveness on new images captured by different sensors or by the same sensor but from different geo-locations. Therefore, it is intractable to find an efficient and accurate land-cover classification method for HRRS images with large diversities.
To characterize the image content of different land-cover categories, many methods investigated the use of spectral and spectral-spatial features to interpret RS images (Jensen and Lulla, 1986; Gong et al., 1992; Casals-Carrasco et al., 2000; Giada et al., 2003; Tarabalka et al., 2010a, b; Zhong et al., 2014; Ma et al., 2017a). However, due to the detailed and structural information brought by the gradually increased spatial resolution, the spectral and spectral-spatial features have difficulty in describing the contextual information contained in the images (Zhao et al., 2016; Zhong et al., 2017; Hu et al., 2016; Yu et al., 2016), which are often essential in depicting land-cover categories in HRRS images. Recently, it has been reported that effective characterization of contextual information in HRRS images can largely improve the classification performance (Shao et al., 2013; Hu et al., 2017; Yang et al., 2015). Among them, deep Convolutional Neural Networks (CNNs) have been drawn much attention in the understanding of HRRS images (Hu et al., 2015a; Zhu et al., 2017), mainly because of their strong capability to depict high-level and semantic aspects of images (Krizhevsky et al., 2012; Zeiler and Fergus, 2014). Currently, various deep models have been adopted to cope with challenging issues in RS image understanding, including e.g. scene classification (Hu et al., 2015a; Xia et al., 2017c), object detection (Xia et al., 2018), image retrieval (Napoletano, 2018; Jiang et al., 2017; Xia et al., 2017b), as well as land-cover classification (Zhao and Du, 2016; Zhao et al., 2015; Zhang et al., 2018a; Maggiori et al., 2017b; Kussul et al., 2017; Volpi and Tuia, 2017).
Nevertheless, there are two main problems in applying deep model to land-cover classification with multi-source HRRS images, which are listed below.
- -
The inadequate transferability of deep learning models: Due to the diverse distributions of objects and spectral shifts caused by the different acquisition conditions of images, deep models trained on a certain set of annotated RS images may not be effective when dealing with images acquired by different sensors or from different geo-locations (Othman et al., 2017). To obtain satisfactory land-cover classification on a RS image of interest, referred as the target image, new specific annotated samples closely related to it are often necessary for model fine-tuning (Maggiori et al., 2017b). Nevertheless, considering that manual annotation requires high labor intensity and is often time-consuming, it is infeasible to label sufficient samples for continuously accumulated multi-source RS images (Lu et al., 2017; Hu et al., 2015b).
- -
The lack of well-annotated large-scale land-cover dataset: The identification capability of CNN models relies heavily on the quality and quantity of the training data (Chakraborty et al., 2015). Up to now, several land-cover datasets have been proposed in the community, and have advanced a lot deep-learning-based land-cover classification approaches (Gerke et al., 2014; Maggiori et al., 2017a; Mattyus et al., 2015). However, the geographic areas covered by most of existing land-cover datasets (Ma et al., 2017b; Gerke et al., 2014; Mattyus et al., 2015) do not exceed and somewhat similar in geographic distributions (Mnih, 2013). The lack of variations in geographic distributions of annotated HRRS images may cause overfitting in model training and limit the generalization ability of learned models. Overall, the insufficient or unqualified training data restrict the availability of deep models for HRRS images.
In this paper, we propose a scheme to adapt deep models to land-cover classification with multi-source HRRS images, which don't have any labeling information. Considering that the textures and structures of the objects are not affected by the spectral shifts, we use contextual information extracted by CNN to automatically mine samples for deep model fine-tuning. Concretely, unlabeled samples in the target image are identified by a CNN model pre-trained on an annotated HRRS dataset, which is referred to as the source data. A subset of them with high confidence are assigned with pseudo-labels and employed to retrieve similar samples from the source data. Finally, the returned results are used to determine whether the pseudo-labels are reliable. In our classification process, a patch-wise classification is initially conducted on the image relying on the multi-scale contextual information extracted by CNN. Then, a hierarchical segmentation is used for obtaining the object boundary information, which is integrated into the patch-wise classification map for accurate results. Specifically, for pre-training CNN models, we annotate 150 Gaofen-2 satellite images to construct a land-cover classification dataset, which is named after Gaofen Image Dataset (GID).
In summary, the contributions of this paper are as follows:
- -
We propose a scheme to train transferable deep models, which enables one to achieve land-cover classification by using unlabeled multi-source RS images with high spatial resolution. In addition, we develop a hybrid land-cover classification that can simultaneously extract accurate category and boundary information of HRRS images. Experiments conducted on multi-source HRRS images, including Gaofen-2, Gaofen-1, Jilin-1, Ziyuan-3, Sentinel-2A, and Google Earth platform data obtain promising results and demonstrate the effectiveness of the proposed scheme.
- -
We present a large-scale land-cover classification dataset, namely GID, which is consist of 150 high-resolution Gaofen-2 images and covers areas more than in China. To our knowledge, GID is the first and largest well-annotated land-cover classification dataset with high-resolution remote sensing images up to 4 m. It can provide the research community a high-quality dataset to advance land-cover classification with HRRS images, like Gaofen-2 imagery.
A preliminary version of this work was presented in (Tong et al., 2018).
The remainder of the paper is organized as follows: In Section 2, we introduce the related works. In Section 3, the introduction of our land-cover classification algorithm is presented. In Section 4, the details and properties of GID coupled with other examined images are described. We present the results of experiments and sensitivity analysis in Section 5 and Section 6, and give the discussion in Section 7. Finally, we conclude our work in Section 8.
Section snippets
Related work
Land-cover Classification: Land-cover classification with RS images aims to associating each pixel in a RS image with a pre-defined land-cover category. To this end, classification approaches using spectral information have been intensively studied. These methods can interpret RS images using the spectral features of individual pixels (Jensen and Lulla, 1986; Gong et al., 1992; Casals-Carrasco et al., 2000), but their performance is often heavily affected by intra-class spectral variations and
Methodology
To efficiently conduct land-cover classification with multi-source HRRS images, we propose a scheme to train transferable deep models, which is pre-trained on labeled land-cover dataset and can be applied to unlabeled HRRS images. Assume that there is a well-annotated large-scale dataset and a newly acquired image without labeling information. We define two domains, called source domain and target domain that are separately associated with the labeled and unlabeled images. Our aim is to
Experimental results
We test our algorithm and analyse the experimental results in this section. Two types of land-cover classification issues are examined: 1) transferring deep models to classify HRRS images captured with the same sensor and under different conditions, 2) transferring deep models to classify multi-source HRRS images. For performance comparison, several object-based land-cover classification methods are utilized. The implementation details, comparison methods, and evaluation metrics are introduced
Sensitivity analysis
In the former section, the experimental results show the promising performance of the proposed method. However, some parameters have impact on the classification results. In this section, we analyse and discuss these factors through additional experiments, including analysis on patch size, segmentation method, and thresholds of transfer learning scheme.
Discussion
Land-cover classification is closely tied to the ecological condition of the Earth's surface and have significant implications for global ecosystem health, water quality, and sustainable land management. Most studies on large-scale land-cover classification generally use the low-/medium-spatial resolution RS images, however, due to the lack of spatial information, these images are insufficient for detailed mapping for high heterogeneous areas (Hu et al., 2013). By contrast, high-spatial
Conclusion
We present a land-cover classification algorithm that can be applied to classify multi-source HRRS images. The proposed algorithm has the following attractive properties: 1) it automatically selects training samples from the target domain based on the contextual information extracted from deep model. In consequence, it does not require new manual annotation or algorithm adjustment when being applied to multi-source images. 2) it uses multi-scale contextual information for classification.
Funding
This work was supported in part by the National Natural Science Foundation of China under Grants 61922065, 61771350, 61871299 and 41820104006, in part by the Open Research Fund of Key Laboratory of Space Utilization, Chinese Academy of Science LSU-SJLY-2017-01, the Outstanding Youth Project of Hubei Province under Contract 2017CFA037.
References (107)
- et al.
Markov random field-based super-resolution mapping for identification of urban trees in vhr images
ISPRS J. Photogrammetry Remote Sens.
(2011) - et al.
Multi-resolution, object-oriented fuzzy analysis of remote sensing data for gis-ready information
ISPRS J. Photogrammetry Remote Sens.
(2004) Object based image analysis for remote sensing
ISPRS J. Photogrammetry Remote Sens.
(2010)- et al.
A multi-scale segmentation/object relationship modelling methodology for landscape analysis
Ecol. Model.
(2003) - et al.
A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using spot-5 hrg imagery
Remote Sens. Environ.
(2012) - et al.
A comparison of spatial feature extraction algorithms for land-use classification with spot hrv data
Remote Sens. Environ.
(1992) - et al.
Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery
Remote Sens. Environ.
(2018) Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks
- et al.
A review of supervised object-based land-cover image classification
ISPRS J. Photogrammetry Remote Sens.
(2017) - et al.
A review of supervised object-based land-cover image classification
ISPRS J. Photogrammetry Remote Sens.
(2017)
Mapping private gardens in urban areas using object-oriented techniques and very high-resolution satellite imagery
Landsc. Urban Plan.
Per-pixel vs. object-based classification of urban land cover extraction using high spatial resolution imagery
Remote Sens. Environ.
Good practices for estimating area and assessing accuracy of land change
Remote Sens. Environ.
A neural network approach using multi-scale textural metrics from very high-resolution panchromatic imagery for urban land-use classification
Remote Sens. Environ.
An evaluation of time-series smoothing algorithms for land-cover classifications using modis-ndvi multi-temporal data
Remote Sens. Environ.
Representative lake water extent mapping at continental scales using multi-temporal landsat-8 imagery
Remote Sens. Environ.
How much does multi-temporal sentinel-2 data improve crop type classification?
Int. J. Appl. Earth Obs. Geoinf.
A hybrid mlp-cnn classifier for very fine resolution remotely sensed image classification
ISPRS J. Photogrammetry Remote Sens.
An object-based convolutional neural network (ocnn) for urban land use classification
Remote Sens. Environ.
How useful is region-based classification of remote sensing images in a deep learning framework?
Segnet: a deep convolutional encoder-decoder architecture for image segmentation
IEEE Trans. Pattern Anal. Mach. Intell.
Classification of hyperspectral data from urban areas based on extended morphological profiles
IEEE Trans. Geosci. Remote Sens.
What's wrong with pixels? some recent developments interfacing remote sensing and gis
GeoBIT/GIS
A multilevel context-based system for classification of very high spatial resolution images
IEEE Trans. Geosci. Remote Sens.
A novel transductive svm for semisupervised classification of remote-sensing images
IEEE Trans. Geosci. Remote Sens.
A novel approach to the selection of spatially invariant features for the classification of hyperspectral images with improved generalization capability
IEEE Trans. Geosci. Remote Sens.
Application of spectral mixture analysis for terrain evaluation studies
Int. J. Remote Sens.
Active batch selection via convex relaxations with guaranteed solution bounds
IEEE Trans. Pattern Anal. Mach. Intell.
Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
IEEE Trans. Pattern Anal. Mach. Intell.
Detection of land-cover transitions in multitemporal remote sensing images with active-learning-based compound classification
IEEE Trans. Geosci. Remote Sens.
Definition of effective training sets for supervised classification of remote sensing images by a novel cost-sensitive active learning method
IEEE Trans. Geosci. Remote Sens.
Imagenet: a large-scale hierarchical image database
Advances in spectral-spatial classification of hyperspectral images
Proc. IEEE
Efficient graph-based image segmentation
Int. J. Comput. Vis.
Borrowing treasures from the wealthy: deep transfer learning through selective joint fine-tuning
Isprs semantic labeling contest
Information extraction from very high resolution satellite imagery over lukole refugee camp, Tanzania
Int. J. Remote Sens.
Semisupervised image classification with laplacian support vector machines
IEEE Geosci. Remote Sens. Lett.
Textural features for image classification
IEEE Trans. on Systems, Man, and Cybernetics
Deep residual learning for image recognition
Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery
Remote Sens.
Fast binary coding for the scene classification of high-resolution remote sensing imagery
Remote Sens.
Deep sparse representations for land-use scene classification in remote sensing images
A comparative study of sampling analysis in the scene classification of optical high-spatial resolution remote sensing imagery
Remote Sens.
Exploring the use of google earth imagery and object-based methods in land use/cover mapping
Remote Sens.
Encoding invariances in remote sensing image classification with svm
IEEE Geosci. Remote Sens. Lett.
Introductory digital image processing: a remote sensing perspective
Geocarto Int.
Retrieving aerial scene images with learned deep image-sketch features
J. Comput. Sci. Technol.
Spatially adaptive classification of land cover with remote sensing data
IEEE Trans. Geosci. Remote Sens.
Imagenet classification with deep convolutional neural networks
Cited by (581)
Classifying raw irregular time series (CRIT) for large area land cover mapping by adapting transformer model
2024, Science of Remote Sensing<sup>Using ZY1-02D satellite hyperspectral remote sensing to monitor landscape diversity and its spatial scaling change in the Yellow River Estuary</sup>
2024, International Journal of Applied Earth Observation and GeoinformationDesign of on-chip coded high-resolution 2D imaging via 3D compressed sensing
2024, Optics and Lasers in EngineeringScale-aware deep reinforcement learning for high resolution remote sensing imagery classification
2024, ISPRS Journal of Photogrammetry and Remote SensingSelf-training guided disentangled adaptation for cross-domain remote sensing image semantic segmentation
2024, International Journal of Applied Earth Observation and Geoinformation