A Prior Semantic Network for Large-Scale Landcover Change of Landsat Imagery

Abstract

Landcover change can reflect changes in the natural environment and the impact of human activities. Remotely sensed big data with large-scale and multi-temporal key characteristics provide the data support for landcover change information extraction. The development of deep learning provides technical method support for information extraction from remotely sensed big data. However, the current mainstream deep learning change detection methods only establish the changing relationship between two phases of images. They cannot directly extract the ground object categories before and after the change. It is easily affected by pseudo-changes caused by the color difference of multi-temporal images, resulting in many false detections. In this paper, we propose a prior semantic network and a difference enhancement block module to establish prior guidance and constraints on changing features to solve the pseudo-change problem. We propose a semantic-change integrated single-task network, which can simultaneously extract multi-temporal landcover classification and landcover change. On the self-made, large-scale multi-temporal Landsat dataset, we have performed multi-temporal landcover change information extraction, reaching an overall accuracy of 83.1% and achieving state-of-the-art performance. Finally, we thoroughly analyzed the landcover change results in the study area from 2005 to 2020.

1. Introduction

Landcover classification from remote sensing images is a significant application in remote sensing [1]. The classification of and changes in landcover can be obtained by using multi-temporal images. Landcover change can intuitively show the dynamic changes of the land surface, which has significant application value in land resource monitoring, ecological protection, urban expansion, returning farmland to forest, etc. [2,3,4,5,6]. However, the multi-temporal landcover classification also has more requirements on the amount of remote sensing data. With the development of remote sensing technology, remote sensing data is developing towards remotely sensed big data [7,8]. Remotely sensed big data with 4V characteristics volume, variety, velocity, and veracity provide massive data for remote sensing information extraction [9,10]. Traditional information extraction methods need to artificially design feature extractors according to data characteristics, such as index-based methods (NDVI, NDWI, NDBI, etc.) [11,12,13], texture-based methods (edge detection, keypoint extraction, etc.) [14,15,16], or statistical-based methods (support vector machines, random forests, etc.) [17,18,19]. When these methods face massive data, due to the limitations of artificial design, it is not easy to design a model that perfectly fits all the data. Therefore, the generalization ability of the model is limited, and the model may not be able to fit the new data. The emergence of deep learning technology has solved the data explosion problem of remotely sensed big data. Deep learning allows the network to independently exploit and learn effective features through deep convolutional neural networks (DCNNs) [20]. The DCNN has a stronger fitting ability to massive data and better generalization ability. At the same time, the deep learning method can quickly and accurately produce large-scale products, significantly saving time, human resources, and financial resources. Therefore, the deep learning method has gradually become the mainstream remote sensing information extraction method [21].

The traditional methods of landcover change detection are mainly based on two ideas [22,23,24,25,26]. The first is to set a change threshold based on the difference between the two images to detect the changing area. This method is largely limited by the quality of image processing. Differences in the colors of the two images are caused by differences in the imaging season and atmospheric conditions. As a result, pseudo-changes may appear in areas that have not changed between the two images. The second is to first classify the landcover of the two phases of images, then make the difference between the two phases of classification results to obtain the changed areas. This method is subject to the landcover classification accuracy for each phase of images. It may also be affected by the color difference between the images of the two phases. There may be errors in the classification results of the two phases, which will also cause pseudo-changes. These pseudo-changes and errors will cause changes in the ground objects to be inconsistent with the real natural scenes. The change detection methods based on deep learning establish a complex correlation between the two images by building a complex DCNN. The network learns the changing area’s features through supervised learning, eliminates the pseudo-changing area’s interference, and finally obtains a more accurate landcover change result.

Change detection methods based on deep learning have made significant development [27]. FC-EF [28] stacks two-phase images into multi-channel images, inputs images into a fully convolutional neural network, and outputs the changed regions. FC-Siam-conc [28] uses two encoders to extract the features of the two-phase images, respectively. It stacks each stage’s features, transfers them to the decoder for feature fusion, and outputs the changed regions. FC-Siam-diff [28] is similar to FC-Siam-conc. The only difference is that the two encoder feature fusion methods are replaced from stacking to difference operation. Similar to FC-EF, CDNet [29] stacks two-phase images into multi-channel images as input. It is a classic encoder–decoder architecture. Based on the dual encoder network, DSIFN [30] adds channel attention and spatial attention mechanisms to improve the detection accuracy of changing regions. UNet++MSOF [31] and DDCNN [32] stack two-phase images as input. The backbone of them adopts the densely connected UNet++ [33] network. These change detection methods can only find the changing area, which can be regarded as a changed/unchanged binary classification problem. For multi-class landcover change detection, the above methods cannot exploit semantic information in the changing area and often need to use additional semantic segmentation networks to supplement the semantic categories. Semantic segmentation in deep learning is pixel-level classification in remote sensing. Since the change detection of landcover requires pixel-by-pixel labels of multi-temporal registered images, there are few practical applications of large-scale landcover change detection based on deep learning methods. However, many studies still indirectly implement the change trend analysis after landcover classification of the multi-temporal images [34]. This paper is completely based on the deep learning method to detect the landcover changes of the multi-temporal images. The category information before and after the change in the ground objects can also be accurately classified.

We call change detection with semantic information before and after the change as semantic change detection. Landcover change is a semantic change detection task. In this paper, we propose a prior semantic network that integrates the difference enhancement block module and compresses the multi-task network into the single-task network, which implements high-precision Landcover change mapping and change details analysis.

In summary, the main contributions of this paper are as follows:

• We propose a prior semantic network architecture. Based on the two-phase data, the third-phase data and labels are introduced as prior constraint knowledge. It can solve the problem of pseudo changes caused by differences in color distribution and greatly improve the stability and robustness of change detection and semantic classification.
• We propose a difference enhancement block module, which weights the differences between the two branches, enhances true changes with large differences, and suppresses pseudo-changes with small differences.
• We compress the multi-task network, which is relatively independent of change detection and semantic segmentation, into a single-task network, which can simultaneously obtain the area of landcover change and the category of ground objects before and after the change in the network output.
• Extensive experiments on our self-made, large-scale, multi-temporal Landsat dataset achieve state-of-the-art performance. Through our proposed network, multi-temporal landcover change detection and specific change trend analysis were carried out for the large-scale study area from 2005 to 2020.

2. Methodology

This chapter mainly introduces the prior semantic network, the difference enhancement block module, and the single-task semantic change integration. We take the two-encoder Siamese UNet as the benchmark network. First, based on Siamese UNet, we add an additional encoder branch as a prior semantic knowledge constraint to build a prior semantic network. Then, the change feature fusion module in the prior semantic network is replaced with a difference enhancement block module to build PSNet-DBB. Finally, we combine the semantic segmentation and change detection multi-task decoders in PSNet into a single-task decoder to build PSNet-ST.

As shown in Figure 1, although the deep learning method can overcome the pseudo-change problem caused by color difference to a certain extent, the semantic information extraction is still unstable. The changing amplitude jitters seriously, and the change in the ground object category does not conform to the actual situation. The pseudo-changes may make it impossible to correctly obtain the change trend of the landcover and lose the practical application value. The current deep learning change detection network only extracts relevant information from the two-phase images, which is greatly affected by the color distribution of the images themselves. Based on the two-phase images, we can introduce another phase of images. The classification labels corresponding to the images of the new phase are also input into the network as auxiliary reference data for semantic information. It can improve the stability of the original two-phase image semantic information extraction, thereby improving the accuracy of landcover changes and ensuring the accuracy of changing trends. Therefore, we propose a prior semantic network architecture to achieve change detection under the constraints of additional reference branches.

Currently, in the mainstream change detection network using dual encoders, the feature fusion of the two branches is mainly performed through concatenation and difference operations. The concatenation operation simply stacks the features without enhancing the changing features. The difference operation expresses the feature difference of the two branches, but it causes the decoder to only have the change features and lose the semantic features. Therefore, we propose a difference enhancement block module, which enhances the features after the feature map difference is weighted to the original feature map as a weight. The module can amplify the obvious change features and suppress the features with very small differences. The reason for this is that such small changes are likely pseudo-changes caused by color differences. The module can also keep the original semantic classification information.

2.1. Prior Semantic Network

UNet is currently the most widely used fully convolutional neural network [35] and is often used as a baseline network in semantic segmentation and change detection. UNet is an encoder–decoder network with simple architecture, fast running speed, and low GPU memory overhead. We also choose UNet as the quasi-baseline network and ResNet-50 as the encoder. However, UNet has only one encoder branch. Two images must be stacked at the input end if the network is used for change detection. The network principle is similar to FC-EF. Therefore, we first add an encoder branch based on UNet, which is also ResNet-50 [36]. The weights of the two encoders are shared to build a Siamese UNet, named SiamUNet. Encoder weight sharing ensures that the feature positions in the two encoder branches are the same so that they are comparable to compute feature differences. The two branches of SiamUNet are fused by a concatenation operation to learn differential features. The network principle is similar to FC-Siam-conc. SiamUNet is the baseline network in this work, and the modules and structures proposed in this paper are gradually added based on the baseline network.

The two encoders of the dual-branch SiamUNet input image data of the T1 and T2 phases, respectively, and then detect the changing area between the T1 and T2 phases. We introduce an additional time-phase T0 of image and label data as prior semantic information, which is input into the network. Since the T0 phase requires label data as the additional input, the number of input channels is one more than that of the T1 and T2 phases. We add a new encoder branch, also ResNet-50, to extract T0 phase features. This branch does not share weights with the T1 and T2 phase branches. We named this branch the prior branch and named the T1 and T2 phase branches as the pre-change branch and post-change branch, respectively.

Unlike the direct fusion of the pre-change branch and post-change branch in SiamUNet, we first directly fuse the prior branch with the pre-change branch. The network builds a complex function map between T0 and T1 images by stacking many convolution operations and eliminates the effects of color differences by itself. The network will pay attention to the change feature information between the T0 and T1 images. With the help of the prior semantic information in the T0 label, the network will automatically establish more accurate semantic information for the T1 phase. The network will learn the differential features before and after the change in ground objects. This differential feature can be regarded as a change mapping feature. The network deduces the ground object category at T1 through T0 prior knowledge and change mapping features in changing areas. It will directly bring the prior semantics of T0 into T1 if there is no change between before and after. We obtain T1 features with T0 prior knowledge, named prior pre-change features. In the same way, we also fuse the prior branch with the post-change branch and establish the change feature association between T0 and T2 and the semantic information of the T2 phase. We obtain the T2 feature with T0 prior knowledge, named prior post-change feature.

Unlike SiamUNet, which can only learn T1 and T2 change features, prior pre-change and prior post-change features can learn the change features and use semantic information to assist in optimizing the change features. It is because the two change features contain the category semantic information of the ground objects, which can eliminate the interference caused by the pseudo-change phenomenon caused by the color difference in the image. In the decoder stage, the prior pre-change and prior post-change features are fused to calculate the changing area. This part of the decoder is called the change task decoder. In addition, the prior pre-change and prior post-change features independently calculate the semantic segmentation results. These two decoders are called segmentation task decoders. The entire network architecture is named the prior semantic network (PSNet), which can implement change detection and semantic segmentation at the same time. It is a multi-task network. The schematic diagram of the network architecture of PSNet is shown in Figure 2.

In the training stage, the images and labels of the T0 phase are stacked into the $N+1$ band data and input to the prior branch. T1-phase images are input to the pre-change branch as N-band data. T2-phase images are input to the post-change branch as N-band data. The pre-change branch shares weights with the post-change branch. The binary change label is used for the loss calculation at the end of the decoder of the change task, and the landcover classification labels of the T1 and T2 phases are respectively used for the loss calculation at the end of the decoder of the two shared weight segmentation tasks. All three loss values guide the backpropagation and gradient update of the network. In the inference stage, we only need to input the images and labels of the T0 phase and the images of the T1 and T2 phases. The changing area of T1 and T2 can be calculated, as well as the respective landcover classification results of T1 and T2.

2.2. Difference Enhancement Block Module

When calculating the changing area for the feature fusion of prior pre-change and prior post-change, if the common difference absolute value method is used to calculate the feature difference, a slight difference in the feature will be regarded as a change. This results in errors and pseudo-changes in the results. Because the features only contain differences, the decoder can only implement the change detection task. At this time, only a multi-decoder multi-task network architecture can be used for the semantic change detection task.

Therefore, we use the concatenation method to fuse the two features containing semantic segmentation information and keep all the feature information completely. At the same time, the square of the difference between the two features is calculated as the weight feature. Then, the fused features containing semantic segmentation information are weighted by the weight feature, which amplifies the changed features and suppresses the pseudo-changed features with minor changes. After the feature difference is squared, when the value is greater than 1, the feature difference weight will be amplified. When the value is less than 1, the feature difference weight will be reduced.

As shown in Figure 3, we first concatenate the prior pre-change and prior post-change features to obtain the fused features. Then we used $1\times 1$ convolution to reduce the number of channels of the fused features by half, the same as the number of channels before fusion. Batch normalization [37] and sigmoid are used to normalize and activate features. Then we calculate the difference square of the prior pre-change and prior post-change features to obtain the difference weight feature. Next, the difference weight feature is weighted to the fusion feature to obtain the difference-enhanced fused feature. Finally, $1\times 1$ convolution, batch normalization, and rectified linear unit (ReLU) are used to reintegrate, normalize, and activate the fused features to obtain the final difference-enhanced feature. We name it the difference enhancement block (DEB) module.

We denote the convolution operation as:

$${\mathrm{W}}^n\left(x\right)={\mathbf{W}}^{n\times n}\odot x+\mathbf{b},$$

(1)

where ⊙ represents the convolution operator, ${\mathbf{W}}^{n\times n}$ represents the $n\times n$ convolutional kernel, $\mathbf{b}$ represents the vector of bias, and x represents the input data.

This section will perform the batch normalization operation after each convolution operation. To simplify the expression, ${\mathrm{W}}^n\left(x\right)$ not only represents the convolution layer but also includes the batch normalization layer. Therefore, the DEB module can be expressed as:

$$f_{DEB}(x_{pre},x_{post})=f_{ReLU}({\mathrm{W}}_2^1((f_{sigmoid}({\mathrm{W}}_1^1(x_{pre}\oplus x_{post})))\otimes {(x_{post}-x_{pre})}^2)),$$

(2)

where ⊕ represents the concatenation operator; ⊗ represents the dot multiply operator; $f_{sigmoid}$ represents the sigmoid function; $f_{ReLU}$ represents the ReLU function; ${\mathrm{W}}_1^1$ and ${\mathrm{W}}_1^2$ represents the first and second $1\times 1$ convolution layer, respectively; $x_{pre}$ represents the prior pre-change feature; and $x_{post}$ represents the prior post-change feature.

We use the DEB module to replace the prior pre-change and prior post-change feature fusion modules in PSNet to build the PSNet-DEB network.

2.3. Single-Task Architecture for Semantic Change

There is certain independence between multiple decoders, which will prevent the features between multi-branches in the decoding stage from directly assisting and optimizing each other in the learning process. As a result, there will be minor contradictions between the semantic segmentation results and the change detection results. For example, the semantic segmentation results of the two phases have not changed, but the change detection results are considered to have changed. The single integrated decoder simultaneously implements semantic segmentation and change detection at the end of the network, which can optimize learning from each other and avoid conflicting problems.

The DEB module in Section 2.2 can highlight the change feature information while keeping the complete semantic segmentation information, which provides a theoretical basis for building a single decoder to directly implement the semantic change task. We remove the two segmentation-task decoders in PSNet-DEB and keep only one change-task decoder. However, we change the output of the change-task decoder from binary-value change to the form of multi-value classification. We choose a number to describe the changing state between every two categories. For example, we label the first class change to the third class as 13, the fourth class change to the second class as 42, and the fifth class remains unchanged as 55. We rename the upgraded decoder as the semantic change decoder and build the PSNet-ST network. The schematic diagram of the network architecture of PSNet-ST is shown in Figure 4.

In the training stage, the input data form of PSNet-ST is the same as the multi-task PSNet. We use the joint label, which can describe the semantic change information, to compute the loss value and guide the network’s backpropagation and gradient updates. In the inference stage, we only need to input the images and labels of the T0 phase and the images of the T1 and T2 phases. Then, we can calculate the categories of the ground object before and after the change from T1 to T2 and deduce the changing area.

3. Experimental Results

3.1. Datasets

We can easily download multi-temporal Landsat imagery, which can be used to study landcover classification and change. However, no public Landsat dataset currently contains both semantic segmentation and change detection labels. Therefore, to test our proposed method’s performance on semantic change detection through experiments, we made a multi-temporal semantic change detection Landsat dataset.

We selected part of central and southern China as the study area, covering an area of 360,000 ${\mathrm{km}}^2$, located between ${110.933}^{\circ }$ E∼${116.622}^{\circ }$ E and ${28.444}^{\circ }$ N∼${34.133}^{\circ }$ N. We downloaded images of five time phases in 2000, 2005, 2010, 2015, and 2020. Each phase needs 26 images to cover the whole research area. The data path is between 121∼126, and the data row is between 36∼40. Among them, the images of 2000, 2005, and 2010 use Landsat-5 data, equipped with a thematic mapper (TM) sensor, including seven bands. Except for the thermal infrared band with a spatial resolution of 120 m, the other bands have a spatial resolution of 30 m. The images of 2015 and 2020 use Landsat-8 data, equipped with the operational land imager (OLI) sensor, including nine bands. Except for the panchromatic band, which has a spatial resolution of 15 m, the other bands have a spatial resolution of 30 m. All downloaded images are at the L1TP level. We only used six bands of data, including blue, green, red, near-infrared, shortwave infrared 1, and shortwave infrared 2. We then mosaicked the downloaded images by year and cropped them according to the latitude and longitude of the study area, removing redundant images outside the study area. Finally, we obtained 20,480 × 20,480 pixels of Landsat image in five phases. The spatial resolution is 30 m.

As shown in Figure 5, to train the semantic change model, we annotated the images from 2000, 2005, and 2010 at the pixel level, including seven categories: woodland, grassland, wetland, waterbody, cultivated land, artificial surface, and bare land. The 2015 and 2020 images shown in Figure 6 are not labeled, and the landcover classification and change results will be inferred through the deep learning method. All labels were visually interpreted in ArcGIS software by a team of 10. Controversial ground objects that cannot be identified on the image are labeled by high-resolution remote sensing images or field surveys. All samples were randomly cross-checked three times, and disputed samples were uniformly determined. Although there is a certain possibility of error in manual labeling, we try our best to minimize it and make the label’s accuracy as close to 100% as possible. High-resolution imagery is the primary reference for the edge of ground objects prone to mislabeling. Using the labels on the high-resolution images to downsample to the medium resolution can eliminate the label errors at the edge of the ground objects. To verify the model’s accuracy more accurately, we randomly selected 1000 points for each category of the ground object in the study area. Then, we obtained the ground truth corresponding to the 2020 images through a field survey and high-resolution image reference.

3.2. Implementation Details

3.2.1. Data Preprocessing

The PSNet network proposed in this paper requires two samples for change detection and one for prior knowledge input. Therefore, we take the sample in 2000 as prior knowledge and the samples in 2005 and 2010 as the data for actual semantic change detection. Since only two samples are needed for the published mainstream change detection network, we use samples in 2005 and 2010 for training to make the evaluation data the same.

We use the sliding window to crop each image with 20,480 × 20,480 pixels into 1600 small tiles with 512 × 512 pixels without overlapping. To evaluate the model’s generalization ability more objectively, we keep a large proportion of the data out of training. Therefore, we divide the dataset into a training set, validation set, and test set according to the ratio of 4:1:5. That is, 640 tiles are used for training the model, 160 tiles are used to validate intermediate model accuracy during the training stage to pick the best model, and 800 tiles do not participate in the training stage and are only used for prediction and accuracy evaluation.

As shown in Figure 7, in addition to keeping semantic segmentation labels, the label data obtain binary change detection samples by comparing two-phase samples with different values. We adopt the form of joint labels to make semantic change labels. We use a two-digit number to denote each pixel’s categories before and after changes. The first digit is the category number before the change, and the second is the category number after the change. When the two digits are the same, there is no change. The category numbers in semantic change map are shown in

Table 1. The category number of semantic changes in joint label. The columns indicate the categories before the change, and the rows indicate the categories after the change.

	Woodland #1	Grassland #2	Wetland #3	Waterbody #4	Cultivated Land #5	Artificial Surface #6	Bare Land #7
Woodland #1	11	12	13	14	15	16	17
Grassland #2	21	22	23	24	25	26	27
Wetland #3	31	32	33	34	35	36	37
Waterbody #4	41	42	43	44	45	46	47
Cultivated land #5	51	52	53	54	55	56	57
Artificial surface #6	61	62	63	64	65	66	67
Bare land #7	71	72	73	74	75	76	77

. The columns indicate the categories before the change, and the rows indicate the categories after the change.

To ensure a more reasonable distribution of data input to the model, we normalize the input data first. The data normalization is defined as:

$$I^{\prime }=\frac{I-mean}{stddev},$$

(3)

where $I^{\prime }$ represents the normalized image data, I represents the original input image data, $mean$ represents the mean value of each band in the image data, and $stddev$ represents the standard deviation value of each band in the image data.

3.2.2. Training Settings

We use the PyTorch deep learning framework [38] to implement the PSNet proposed in this paper and other mainstream change detection networks published. We used four NVIDIA RTX 3090 GPUs with 24 GB memory to train the model. Data augmentation operations include random horizontal flips, random vertical flips, and random rotation. The optimizer is AdamW [39], and the batch size is set to 32. The initial learning rate is $1\times {10}^{-5}$ and gradually increases to $1\times {10}^{-3}$ during the first 10 epochs. Then, the learning rate is automatically adjusted using the model validation accuracy. When the accuracy has not improved for 20 consecutive epochs, the learning rate is multiplied by the drop coefficient of 0.3. When the learning rate drops to $1\times {10}^{-7}$, the training process ends.

The formula of the early learning rate increasing stage is:

$$lr={lr}_0·{\left(\frac{{lr}^{*}}{{lr}_0}\right)}^{\frac{t}{k·e}},$$

(4)

where $lr$ represents the real-time learning rate, ${lr}_0$ represents the initial learning rate, ${lr}^{*}$ represents the maximum learning rate, t represents the real-time training iterations, k represents the number of iterations per epoch, and e represents the number of training epochs when the learning rate reaches the maximum value.

The formula of the later learning rate automatic decreasing stage is:

$${lr}^{\prime }=\alpha ·lr,$$

(5)

where ${lr}^{\prime }$ represents the decreased learning rate, $lr$ represents the learning rate before decreasing, and $\alpha$ represents the drop coefficient.

The cross entropy loss can optimize the model by pixel level. Lovász-softmax [40] loss can optimize the model by region level and from intra-class and inter-class differences. Therefore, we choose cross-entropy loss and Lovász-softmax loss as loss functions to train the network. The binarized version corresponding to the two losses is selected in the binary change detection task. The task loss is calculated as follows:

$$L=L_{ce}+L_{Lov\acute{a}sz}$$

(6)

For the single-task semantic change network PSNet-ST, Equation (6) is the final loss value. For multi-task PSNet, multiple branch losses need to be added according to Equation (7):

$$L_{MT}=L_{change}+L_{T1}+L_{T2}$$

(7)

3.2.3. Evaluation Metrics

We mainly adopt three evaluation metrics: overall accuracy (OA), intersection over union (IoU), and F1 score. OA represents the proportion of correctly classified pixels among all pixels. IoU is used to evaluate the accuracy of a certain class, where intersection refers to the number of correctly classified pixels, and union refers to the sum of the number of correctly classified and misclassified pixels. The F1 score is also used to evaluate the accuracy of a certain class, taking into account both precision and recall.

We denote all pixels according to the following rules: $TP$ means the label is true and the prediction is true. $FP$ means the label is false and the prediction is true. $FN$ means the label is true and the prediction is false. $TN$ means the label is false and the prediction is false.

The formula for OA is as follows:

$$OA=\frac{TP+TN}{TP+FP+FN+TN}$$

(8)

The formula for IoU is as follows:

$$IoU=\frac{TP}{TP+FP+FN}$$

(9)

F1 score is calculated by $precision$ and $recall$:

$$F1=2·\frac{precision·recall}{precision+recall},$$

(10)

where $precision$ and $recall$ are calculated by $TP$, $FP$, and $FN$:

$$precision=\frac{TP}{TP+FP},$$

(11)

$$recall=\frac{TP}{TP+FN}.$$

(12)

For multi-task PSNet, the binary change detection accuracy is evaluated with IoU and F1 scores. The semantic segmentation task evaluates single-class accuracy using the F1 score and overall accuracy using mean F1 score (mF1) and OA. Semantic change detection accuracy for single-task PSNet-ST using F1 score was used to evaluate the accuracy of each change.

3.3. Comparing Methods for Binary Change Detection

Since other published mainstream change detection networks can only implement binary change detection, to verify the effect of the prior semantic branch and DEB module proposed in this paper, we use the PSNet series for binary change detection tasks for a fair comparison.

Table 2. The quantitative binary change detection accuracy comparison between the PSNet series network and other mainstream change detection networks.

Methods	IoU	F1
FC-EF	21.81	32.25
FC-Siam-diff	13.36	16.61
FC-Siam-conc	22.01	32.60
DSIFN	29.16	44.81
DDCNN	16.99	23.49
CDNet	25.08	37.95
UNet++MSOF	23.44	35.11
SiamUNet	31.77	49.06
PSNet	44.01	62.79
PSNet-DEB	51.87	68.62
PSNet-ST	65.75	80.91

shows the quantitative accuracy comparison of mainstream change detection networks and the PSNet series used in this paper. It can be seen that the F1 score of the mainstream change detection network can only reach 44.81%. The accuracy of most networks can exceed 30%, and the accuracy of FC-Siam-diff and DDCNN is lower. The baseline SiamUNet built based on the idea of UNet and Siamese encoder in this paper can reach 49.06%. A prior branch is added to the baseline, and the images and labels of the prior reference phase are used for additional constraints. The accuracy of the PSNet can reach 62.79%. It can be seen that prior knowledge is very effective in improving accuracy. Based on PSNet, we replace the multi-branch encoder feature fusion module with the DEB module proposed in this paper. The DEB module performs filtering, weighting, and optimizing the changing features. The PSNet-DEB network achieves 68.62% accuracy. Therefore, using feature differences to weight, optimize and fuse the features has a certain effect on improving accuracy. Finally, based on PSNet-DEB, the semantic segmentation and change detection decoder are combined to build PSNet-ST. The single-task decoder is used to directly learn the categories of ground objects before and after the change. Since the two tasks can be optimized for each other after the decoder is integrated, the accuracy of change detection is greatly improved, reaching 80.91%.

We visualize the change detection results of the mainstream change detection network and the PSNet series network proposed in this paper. Figure 8 shows comparison charts of the change detection results. In the first group, it can be seen that the errors of the three methods, FC-Siam-diff, DDCNN, and UNet++MSOF, are very obvious, and the change detection fails. The error of the CDNet result is also more conspicuous. FC-EF, FC-Siam-conc, and DSIFN missed changes in waterbodies. SiamUNet can detect changes in waterbodies, but there are many false detections. After adding prior knowledge constraints, PSNet can reduce some false detections. After using the DEB module, the waterbody changes detected by PSNet-DEB are more accurate. However, many small changes are missed since DEB modules inhibit small changes, and multi-task decoders cannot directly assist each other in the learning stage. For the single-task network PSNet-ST after multi-decoder integration, the change results of waterbody are very accurate, and other small changing objects can also be detected smoothly.

In the second group, the errors of FC-Siam-diff, UNet++MSOF, and DDCNN are very obvious. It can be seen from the images that there is no major change in the two phases of the ground objects. However, due to the impact of imaging conditions and seasonal factors, the cultivated land shows completely different colors in the images of the two phases. It also brings more significant challenges to change detection. Without the constraints of prior knowledge, baseline SiamUNet, like other mainstream networks, has many false detections. While the prior knowledge assists PSNet in reducing the false detection rate, the DEB module further eliminates small patches with false detections. After the PSNet-ST integrated decoders, the changes in the waterbody can be correctly detected, and the very small changes that have been eliminated in PSNet-DEB can also be successfully detected. More comparisons can be found in Appendix A.1.

To sum up, FC-Siam-diff and DDCNN almost completely fail for change detection on 30 m resolution Landsat images. FC-EF and FC-Siam-conc mainly show more missed detections, while DSIFN, CDNet, and UNet++MSOF show more false detections. In our proposed method, the effect of adding one phase image as prior knowledge is pronounced. Although some false detections exist, the detected change contours are gradually approaching the labels. The DEB module can suppress a large number of false detections. However, semantic segmentation and change detection tasks are independent of each other. The semantic segmentation results cannot be used to optimize the change detection results. Therefore, there will be over-suppression, and small change areas will be missed. After the semantic segmentation and change detection tasks are combined into a single task, the features are optimized for each other. The advantages of the prior semantic information and the DEB module are integrated, and the shortcomings are overcome. The best change detection performance is achieved.

3.4. Comparing Methods for Semantic Change Detection

Since other mainstream change detection networks cannot achieve change detection tasks with semantic information, we only compare the baseline SiamUNet, PSNet, PSNet-DEB, and PSNet-ST networks. SiamUNet has no prior knowledge constraints, and the two decoders for semantic segmentation and change detection tasks are independent and belong to a multi-task network. PSNet has prior knowledge constraints, the two decoders of semantic segmentation and change detection tasks are separated, and it is also a multi-task network. PSNet-DEB is basically the same as PSNet, except that the DEB module replaces the multi-branch feature fusion module, and the feature difference is used for feature weighting enhancement. PSNet-ST has prior knowledge constraints and the DEB module for feature optimization. The single decoder implements the simultaneous extraction of the ground object categories of the two phases. Therefore, semantic segmentation and change detection tasks can be performed simultaneously, which belongs to a single-task network.

Table 3. The quantitative landcover classification accuracy comparison between the SiamUNet and the PSNet series network.

Method	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Mean F1	OA
SiamUNet	87.46	38.18	70.30	78.22	86.47	56.62	33.20	64.35	83.45
PSNet	90.32	77.62	75.27	88.88	94.07	83.10	70.66	82.84	92.77
PSNet-DEB	94.64	77.11	82.94	89.12	94.99	83.71	74.55	85.29	93.21
PSNet-ST	96.23	82.49	90.88	91.51	95.33	86.04	77.49	88.56	94.26

shows the quantitative accuracy comparison of landcover classification between SiamUNet and PSNet series networks. The OA of baseline SiamUNet is 83.45%, and the accuracy of grassland, artificial surface, and bare land is relatively low. After adding prior knowledge constraints, the OA of PSNet reaches 92.77%, and the accuracy of each category is significantly improved. After the DEB module is integrated with the multi-decoder, the accuracy is further improved. The OA of the final single-task network PSNet-ST can reach 94.26%.

Table 4. The quantitative accuracy comparison with ground truth in 2020 between the SiamUNet and the PSNet series network.

Method	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Mean F1	OA
SiamUNet	63.24	13.89	54.37	61.41	70.15	32.83	16.92	44.69	63.57
PSNet	84.34	72.43	70.79	83.57	88.29	78.49	65.77	77.67	88.81
PSNet-DEB	89.66	72.57	78.19	84.55	89.54	78.26	69.63	80.34	91.20
PSNet-ST	91.82	78.39	85.90	86.10	90.94	81.15	72.29	83.80	92.99

shows the accuracy comparison of SiamUNet and PSNet series network with ground truth in 2020. All methods showed a downward trend in accuracy. Without prior knowledge constraints, SiamUNet has the most severe drop in accuracy. The PSNet series network has only a slight decrease in accuracy, which shows that with the prior knowledge constraints, the network’s generalization ability has been significantly improved. The OA of the best-performance network PSNet-ST can reach 92.99%, which means that out of 7000 ground truth points, 6509 points are correctly classified.

For the quantitative accuracy evaluation of semantic change detection, we construct a matrix indicating the mutual conversion to describe the change accuracy between any two categories. As shown in

Table 5. The semantic change detection accuracy of SiamUNet. (Metric: F1 score).

	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Mean Out
Woodland	87.21	62.42	59.17	62.85	65.29	65.90	61.66	66.36
Grassland	62.35	38.25	89.92	59.67	61.64	59.46	59.84	61.59
Wetland	59.05	59.00	70.55	67.81	63.75	60.31	61.00	63.07
Waterbody	62.62	60.07	61.82	78.51	65.63	65.18	61.13	65.00
Cultivated Land	64.82	61.73	63.83	67.23	85.68	74.80	63.13	68.75
Artificial Surface	59.66	59.00	59.41	61.98	64.39	55.08	59.84	59.91
Bare Land	60.76	59.09	59.79	62.52	60.34	66.54	34.33	57.62
Mean In	65.21	57.08	66.36	65.79	66.67	63.89	57.28	63.18

, the column represents the object category before the change, and the Mean Out column represents the mean F1 score calculated based on the category before the change. The row represents the object category after the change, and the Mean In row represents the mean F1 score calculated based on the category after the change. Diagonal elements indicate that the ground object has not changed. Table 5, shows the semantic change detection accuracy between any two categories in the results of SiamUNet, and the overall mean F1 score is 63.1%.

Table 6. The semantic change detection accuracy of PSNet. (Metric: F1 score).

	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Mean Out
Woodland	97.00	71.22	69.05	71.50	72.11	71.52	72.56	74.99
Grassland	73.85	83.29	69.10	70.65	72.60	70.71	71.15	73.05
Wetland	69.20	69.03	65.05	72.64	70.66	69.84	70.07	69.50
Waterbody	71.69	69.77	70.34	92.21	72.88	72.49	72.10	74.50
Cultivated Land	72.12	70.95	69.24	74.14	96.01	77.34	71.60	75.91
Artificial Surface	70.55	69.83	69.19	70.97	71.83	85.73	70.18	72.61
Bare Land	71.17	70.80	69.28	71.53	70.23	71.06	81.07	72.16
Mean In	75.08	72.13	68.75	74.81	75.19	74.10	72.68	73.25

shows the semantic change detection accuracy between any two categories in the results of PSNet, and the overall mean F1 score is 73.25%.

Table 7. The semantic change detection accuracy of PSNet-DEB. (Metric: F1 score).

	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Mean Out
Woodland	96.35	90.12	60.00	86.54	75.69	82.13	86.39	82.46
Grassland	86.68	83.32	60.00	75.97	86.42	73.72	60.00	75.16
Wetland	81.52	60.00	92.88	84.28	84.47	75.01	60.00	76.88
Waterbody	87.45	75.83	86.74	92.07	80.02	85.03	80.83	84.00
Cultivated Land	76.76	87.13	84.76	86.35	95.15	86.40	81.49	85.43
Artificial Surface	85.03	77.41	83.61	85.99	75.95	83.75	60.00	78.82
Bare Land	82.68	60.00	60.00	82.05	77.72	60.00	72.52	70.71
Mean In	85.21	76.26	75.43	84.75	82.20	78.01	71.61	79.07

shows the semantic change detection accuracy between any two categories in the results of PSNet-DEB, and the overall mean F1 score is 79.07%.

Table 8. The semantic change detection accuracy of PSNet-ST. (Metric: F1 score).

	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Mean Out
Woodland	97.28	82.52	80.70	81.68	82.46	81.43	82.28	84.05
Grassland	82.66	87.58	79.15	81.92	82.31	80.20	82.14	82.28
Wetland	84.26	81.17	89.37	81.50	82.50	83.58	79.70	83.15
Waterbody	82.25	80.86	80.58	94.14	82.68	83.74	81.20	83.64
Cultivated Land	82.53	81.83	80.85	84.38	96.46	87.50	81.96	85.07
Artificial Surface	81.29	79.85	79.99	80.88	81.81	87.67	82.33	81.97
Bare Land	81.47	81.63	79.17	81.64	80.94	82.42	83.59	81.55
Mean In	84.53	82.20	81.40	83.73	84.16	83.79	81.89	83.10

shows the semantic change detection accuracy between any two categories in the results of PSNet-ST, and the overall mean F1 score is 83.10%. It can be seen that prior constraint knowledge, DEB module, and single-task integration can significantly improve the semantic change detection task.

We visualize the semantic change detection results of baseline SiamUNet and the PSNet series networks proposed in this paper. Figure 9 is a comparison chart of the semantic change detection results. In each set of examples, the first line is the landcover classification results in the pre-change phase. The second line is the classification results before the change in the changing area. The third line is the classification results after the change in the changing area. The fourth line is the landcover classification results in the post-change phase.

In the first group, the main change is that the cultivated land becomes the waterbody, with some other minor changes. SiamUNet’s landcover classification results are not detailed enough, and small objects are missed. Due to false detections in the change detection, there are errors in the changing area that the before-and-after phases do not in fact change. PSNet has fewer false detections with the help of prior knowledge constraints. However, due to the impact of multi-task independent decoders, the change detection result contradicts the result of semantic segmentation. That is, the change detection branch believes there has been a change, and the semantic segmentation branch believes the ground objects in the before-and-after phases are the same. Smaller fragmented changes are missed from the PSNet-DEB results. PSNet-ST has excellent landcover classification results and change results. After integrating the multi-task decoder into a single-task decoder, the inconsistency between the two results has been eliminated.

In the second group, similar to the first group, the classification results of SiamUNet’s landcover are not detailed enough, the change detection error is obvious, and a large number of unchanged ground objects are placed in the changing area. With the help of prior knowledge constraints, PSNet has fewer false detections, and the changing area is still too large. The result of PSNet-DEB is close to the label, but the bare land’s change in the middle of the image is fragmented and missed. PSNet-ST completely extracts the undetected bare land in PSNet-DEB. It can all be extracted, whether it is a large or a small change. The landcover classification results of the two phases are also very accurate. More comparisons can be found in Appendix A.2.

In summary, the prior semantic knowledge constraints, DEB module, and single-task integrated decoder strategy proposed in this paper have achieved state-of-the-art performance in Landsat’s semantic change detection task.

4. Discussion

In this section, based on the best-performing PSNet-ST model in Section 3, we perform semantic change detection on the four-phase images of the study area in 2005, 2010, 2015, and 2020 and obtain large-scale landcover change results. In addition, the samples in 2000 are used as prior knowledge to assist and constrain the other two phases of data for training and prediction. In addition to this section, more details on these results can be found in Appendix B.

To count the process state of the mutual changes between the ground objects, we adopted a category transition matrix to represent the mutual change areas. As shown in

Table 9. Landcover change statistics from 2005 to 2010. (Unit: square kilometers).

	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Total Out
Woodland	118,172.30	425.12	7.38	266.38	3978.30	402.38	51.21	5130.78
Grassland	423.47	3633.26	0.49	14.13	210.93	11.00	3.84	663.87
Wetland	14.87	1.96	2281.03	403.33	74.09	22.80	1.55	518.59
Waterbody	286.05	11.38	129.09	25,023.67	1560.86	349.48	20.33	2357.17
Cultivated Land	4105.40	206.11	56.00	1637.30	181,559.61	4757.88	57.84	10,820.53
Artificial Surface	294.72	7.63	14.76	248.93	3081.54	22,843.81	19.37	3666.94
Bare Land	55.41	2.34	0.10	34.79	63.18	24.71	635.28	180.52
Total In	5179.92	654.53	207.83	2604.85	8968.90	5568.25	154.14	23,338.41
Total Change	49.13	−9.34	−310.76	247.68	−1851.63	1901.31	−26.38	-

, the column represents the area of a certain category becoming other categories. The Total Out column represents the total area of a certain category turning into other categories, which can be regarded as the transfer-out. The row represents the area of each other category turned into a certain category. The Total In row represents the total area of other categories turned into a certain category, which can be regarded as the transfer-in. The Total Change row represents the overall area change of each category, which is calculated by combining the transferred-out and transferred-in areas of the category.

4.1. Analysis of Landcover Change from 2005 to 2010

The landcover change in the whole study area from 2005 to 2010 is shown in Figure 10. The figure shows the landcover classification results in 2005, the landcover classification results in 2010, and the two-phase corresponding ground object classes in the changing area. Equivalent to the ground objects in Figure 10e becomes the ground objects in Figure 10f. Table 9 is the category transition matrix of the two-phase landcover changes. It can be seen that from 2005 to 2010 in the study area, the woodland, grassland, and bare land changed very little. The wetland area becomes smaller, and the water body area increases. The cultivated land area decreased more, while the artificial surface area increased more. A more specific analysis shows a mutual exchange between woodland and cultivated land. It is caused by the interaction between returning farmland to forest and cutting down trees for reclamation. As the main feature of greening, the woodland area remains unchanged. There is less exchange between grassland and cultivated land, as deserted arable land grows weeds, which can also be reclaimed for cultivation. There is a mutual exchange between wetlands and waterbodies. This is due to the similarities between wetlands and waterbodies. Shallow tidal flats submerged by water will be classified as wetlands. Affected by the imaging season, there will be a mutual conversion between wetlands and waterbodies. There is also a small exchange between waterbodies and cultivated land. This is because when there is more water storage in paddy fields, it looks similar to waterbodies. Since urban development is on a trend of continuous expansion, artificial surfaces occupy more cultivated land. The cultivated land around the city is changed to artificial surfaces. With urbanization, the rural population and the area of cultivated land decrease, and the migration of the rural population to cities will make urban expansion a usual trend. Some villages were demolished to build new reservoirs. However, some of the demolished village lands were planted with trees and converted into woodland. At the same time, some urban artificial surfaces have been re-planned as forest parks. Therefore, in five years, many artificial surfaces have been converted into waterbodies and woodland. More changes’ details can be found in Appendix B.1.

4.2. Analysis of Landcover Change from 2010 to 2015

The landcover change in the whole study area from 2010 to 2015 is shown in Figure 11. The figure shows the landcover classification results in 2010, the landcover classification results in 2015, and the two-phase corresponding ground object classes in the changing area. The ground objects in Figure 11e become equivalent to the ground objects in Figure 11f.

Table 10. Landcover change statistics from 2010 to 2015. (Unit: square kilometers).

	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Total Out
Woodland	122,209.50	89.21	0.31	57.79	847.25	122.75	25.41	1142.71
Grassland	157.05	4061.13	0.20	5.83	59.67	3.28	0.62	226.66
Wetland	9.91	0.53	2419.57	39.37	10.83	8.63	0.03	69.29
Waterbody	81.57	3.66	27.43	26,873.41	486.73	147.75	7.97	755.11
Cultivated Land	965.14	55.31	15.53	462.25	186,001.79	3004.00	24.49	4526.72
Artificial Surface	42.62	1.60	1.87	28.02	374.65	27,957.45	5.84	454.60
Bare Land	2.51	0.34	0.03	1.41	2.29	7.56	775.28	14.14
Total In	1258.79	150.64	45.37	594.68	1781.42	3293.97	64.36	7189.22
Total Change	116.08	−76.02	−23.92	−160.43	−2745.30	2839.37	50.22	-

is the category transition matrix of the two-phase landcover changes. It can be seen that from 2010 to 2015, the changes in grassland, wetland, and bare land were very small in the study area. The changes in woodlands and waterbodies are also not obvious enough. The area of cultivated land decreased more, while the artificial surface area increased more. A more specific analysis shows a mutual exchange between woodland and cultivated land. The mutual exchange between the waterbody and the cultivated land is also similar to the last five years because when there is more water in a paddy field, it looks similar to a waterbody. Cultivated land continues to be transformed into artificial surface, indicating that urbanization further devours the surrounding cultivated land. Compared with the last five-year changes, the area of cultivated land has decreased more, and the artificial surface area has also increased. This shows that the economy has developed faster in the past five years. The speed of urbanization has also accelerated. More details on these changes can be found in Appendix B.2.

4.3. Analysis of Landcover Change from 2015 to 2020

The landcover change in the whole study area from 2015 to 2020 is shown in Figure 12. The figure shows the landcover classification results in 2015, the landcover classification results in 2020, and the two-phase corresponding ground object classes in the changing area. The ground objects in Figure 12e become equivalent to the ground objects in Figure 12f.

Table 11. Landcover change statistics from 2015 to 2020. (Unit: square kilometers).

	Woodland	Grassland	Wetland	Waterbody	Cultivated Land	Artificial Surface	Bare Land	Total Out
Woodland	122,767.65	39.60	0.20	47.50	456.40	139.46	17.48	700.65
Grassland	119.15	4060.60	0.04	3.76	25.92	1.97	0.31	151.16
Wetland	5.78	0.14	2412.62	27.13	8.30	10.89	0.07	52.32
Waterbody	52.21	2.34	18.14	26,887.04	353.20	150.39	4.76	581.04
Cultivated Land	572.89	46.89	7.24	424.72	183,451.63	3264.78	15.06	4331.58
Artificial Surface	16.90	1.46	0.78	18.02	189.90	31,021.68	2.67	229.74
Bare Land	1.12	0.24	0.02	1.05	1.00	5.39	830.81	8.83
Total In	768.05	90.68	26.42	522.20	1034.73	3572.89	40.36	6055.33
Total Change	67.40	−60.49	−25.90	−58.85	−3296.85	3343.15	31.53	-

is the category transition matrix of the two-phase landcover changes. It can be seen that from 2015 to 2020 in the study area, except for cultivated land and artificial surface, the area changes of other categories are very small. The area of cultivated land was significantly reduced, and the artificial surface area was significantly increased. In a more specific analysis, there is a small interchange between woodland and cultivated land. This is because there will be a small dynamic balance change at the boundaries. In addition, a small part of the woodland has been turned into artificial surface. The reason for this is the encroachment of some forest land by urban development. Wetlands are relatively stable, indicating that wetland protection policies have achieved practical results. The exchange of cultivated land and wetlands is also due to the similarity between paddy fields and waterbodies. A large area of cultivated land has become artificial surface, and the change is larger than in the previous ten years. It shows that the city expanded very rapidly from 2015 to 2020, occupying a large amount of cultivated land around the city, reflecting the acceleration of urbanization and the rapid development of the economic level. More details on these changes can be found in Appendix B.3.

4.4. Implications and Limitations

The PSNet proposed in this paper solves three critical problems encountered by the multi-temporal Landsat landcover changes. The first problem is the pseudo-changes caused by differences in color distribution. The second problem is the enhancement and suppression of true and false changes. The third problem is that multi-task networks cannot jointly optimize and constrain each other when performing change detection and semantic segmentation.

For the study of landcover changes, independent semantic segmentation of multi-temporal images is a mainstream method. This method is very sensitive to color differences between multi-temporal images, resulting in many pseudo-change errors in the results. However, the current method of remote sensing multi-temporal image change detection can only focus on the binary information of change and unchanged. It cannot obtain information on the mutual change process between categories. Therefore, a new idea is proposed to design a two-in-one single-task network for semantic segmentation and change detection to solve these three critical problems.

We add an additional encoder branch to the mainstream Siamese network for remote sensing change detection. The original two Siamese branches extract the image features of the two phases, respectively. Then, the data and labels of the third phase are introduced into the newly added additional encoder, which is used as prior knowledge to guide and constrain the feature learning of the original two Siamese branches. Under the constraints of prior knowledge, the two Siamese encoders are simplified from learning complete texture features to only learning their change information relative to the prior image. In the region that has not changed, the label of the a priori phase is directly brought in. Since the network learns two-phase change thresholds based on samples, this dramatically reduces the problem of pseudo-changes caused by color differences. At this time, PSNet adds two additional decoders to implement the multi-task semantic change.

In the Landsat image with a resolution of 30 m, most of the changed features are very small, maybe only one pixel wide. As a result, the distinction between true changes and false changes is not high enough. We redesigned the commonly used difference or concatenation operation and used the difference square to amplify the true change and reduce the false change. The optimized features are then weighted onto feature maps with complete semantic information. This way, pseudo-change errors that are difficult to eliminate can be suppressed. The kept complete semantic features can lay the foundation for the subsequent semantic change two-in-one single-task network.

When using multiple decoder branches to implement the tasks of change detection and semantic segmentation, only the features of the shallow encoder will be fused. However, the decoders are still relatively independent. Therefore, the multi-branch features at the end of the network cannot constrain each other for optimization, information cannot be shared, and even contradictory errors may occur. We combine multiple decoders into a single decoder that converts two-phase semantic segmentation samples into a single semantic change sample with joint labels. With the help of feature learning enhancement and change feature optimization of the prior knowledge branch and DEB module, the single-task network directly implements semantic change detection in one step and can learn complex change states. While improving the efficiency of multi-task learning, the single-task network also solves the risk of conflicting multiple decoders, significantly improving the accuracy and reliability of semantic change detection.

The change detection method with complete semantic information provides powerful support for multi-temporal landcover changes. The mutual change information extracted by PSNet provides data support for analyzing landcover classification and change. Based on the landcover change data, it is possible to further discover and explore phenomena, such as returning farmland to forests, wetland ecological protection, and urban expansion, and mine information, such as the status quo of social and economic development and policy decision making and planning. The PSNet proposed in this paper can quickly extract and analyze the landcover change information in large-scale and multi-temporal dimensions and has significant application value. Based on PSNet, we also conduct a detailed analysis of the large-scale multi-temporal Landsat landcover changes from 2005 to 2020. Urban expansion and the devouring of cultivated land are the most critical keywords obtained in the land cover change analysis.

Compared with mainstream semantic segmentation and change detection networks, the PSNet proposed in this paper provides a new problem-solving idea. The idea is the introduction of prior knowledge constraints, which significantly improves the accuracy and has more practical application value. However, this also brings certain limitations. When performing semantic change detection on two-phase images, the third-phase images and labels must be sacrificed. That is to say, at least three registered multi-temporal samples are required for model training. However, the more time phases there are, the more difficult it is to label samples. Therefore, while PSNet achieves higher accuracy, the workload of manual interpretation in the early stage is also higher than other mainstream algorithms. In future research, we will explore unpaired labels as prior knowledge to improve network accuracy, which may avoid manual labeling work in additional phases and reduce the workload and difficulty of preparatory work. We will also apply our proposed PSNet to single-band images and high-resolution images.

5. Conclusions

In this paper, we propose a prior semantic network for the Landsat semantic change task. Based on the dual-branch Siamese network, we add a prior knowledge encoder branch to solve the problem of pseudo-changes caused by color distribution differences. We design a difference enhancement block module to replace the common difference or concatenation operation and solve the problem of the enhancement and suppression of true and false changes. We propose a single-task PSNet, which combines multiple decoders into one decoder, solving the problem that features cannot be jointly optimized and mutually constrained in multi-task networks. Our proposed method achieves state-of-the-art performance on large-scale multi-temporal Landsat landcover change datasets, far exceeding other change detection and semantic segmentation networks. Based on PSNet, we conducted a specific analysis and discussion on the landcover changes in an area of central and southern China. The acceleration of urbanization construction and the acceleration of economic development are the two keywords we have found from the results of landcover changes over 15 years. This work has a particular value for sustainable development goals. In future research, we will study constraint learning on unpaired samples and generalize PSNet to more multi-temporal remote sensing data of different resolutions.