Purifying SLIC Superpixels to Optimize Superpixel-Based Classification of High Spatial Resolution Remote Sensing Image

Abstract

Fast and accurate classification of high spatial resolution remote sensing image is important for many applications. The usage of superpixels in classification has been proposed to accelerate the speed of classification. However, although most superpixels only contain pixels from single class, there are still some mixed superpixels, which mostly locate near the edge of different classes, and contain pixels from more than one class. Such mixed superpixels will cause misclassification regardless of classification methods used. In this paper, a superpixels purification algorithm based on color quantization is proposed to purify mixed Simple Linear Iterative Clustering (SLIC) superpixels. After purifying, the mixed SLIC superpixel will be separated into smaller superpixels. These smaller superpixels are pure superpixels which only contain a single kind of ground object. The experiments on images from the dataset BSDS500 show that the purified SLIC superpixels outperform the original SLIC superpixels on three segmentation evaluation metrics. With the purified SLIC superpixels, a classification scheme in which only edge superpixels are selected to be purified is proposed. The strategy of purifying edge superpixels not only improves the efficiency of the algorithm, but also improves the accuracy of the classification. The experiments on a remote sensing image from WorldView-2 satellite demonstrate that purified SLIC superpixels at all scales can generate classification result with higher accuracy than original SLIC superpixels, especially at the scale of $20\times 20$, for which the accuracy increase is higher than 4%.

1. Introduction

In recent decades, the available technologies for Earth observation generate a lot of high spatial resolution airplane and satellite images. However, the huge number of generated remote sensing image data needs to be processed and analyzed for various purposes [1,2,3]. The classification of high spatial resolution remote sensing data is a crucial step for processing these data. Nonetheless, it is still a challenge to classify the data accurately and efficiently in practical applications.

In the high spatial resolution remote sensing images, the spatial information is abundant and the spectral information is complex [4]. Therefore, the traditional pixel based classification algorithm that cannot make full use of the spatial information is not the best choice for the classification task. The object based image analysis (OBIA), which replaces the pixel with the object containing a group of connected homogeneous pixels, has been developed to better exploit the spatial information in the image [4,5,6]. The object based classification has a lot of advantages over the pixel based classification in dealing with high spatial resolution imagery [7]. However, the performance of object based classification depends on the segmentation quality. Many segmentation algorithms have been proposed, but segmentation is still a problem in OBIA [5,8,9]. Many segmentation algorithms including the popular multiresolution segmentation [10] have trouble in finding optimal parameters [9,11]. Even objects from the same class may present in different scales. If the scale parameter is too large, under-segmented objects, which negatively influence the classification, will be generated. Therefore, the classification based on over-segmented objects can be a viable alternative to objects, even though the over-segmented objects cannot make good use of the shape information.

Superpixel is a good choice for over-segmentation [12,13]. It was first introduced as a preprocessing step to over-segment an image into small superpixels for simplifying the computation in subsequent stages [14]. The superpixel can carry more information than pixels and adhere better to the natural image boundaries [15,16]. In addition, superpixels can decrease the computation complexity and speed up the subsequent processing [12,13,14]. Moreover, superpixels reduce the susceptibility to noise and outliers, capturing image redundancy [17]. In the field of remote sensing image processing, superpixels have been applied in a diverse range of applications and data types [18,19,20,21,22,23,24,25]. In [13], a classification framework using a superpixel-based graphical model to capture the contextual information and the spatial dependence between the superpixels is proposed. In [26], superpixels were used to define the neighborhood similarity of a pixel adapted to the local characteristics of each image. In [27], a scheme for superpixel description based on Bag of visual Words, which includes information from adjacent superpixels, is introduced. In [28], edge-based processing and superpixel processing were combined for automatic generation of thematic maps for small agricultural parcels. In [29], the Superpixel Contour algorithm was introduced to generate a set of different levels of segmentation for a semi-automatic optimization of object-based classification of multitemporal data. In [1], superpixels were applied in the segmentation and classification of very high resolution remote sensing data.

Many algorithms for generating superpixels have been developed [30]. Among them, six algorithms [12,31,32,33,34,35] were recommended in [30] because these algorithms can be considered stable according to their superior performance regarding boundary recall, under-segmentation error and explained variation. Contour relaxed superpixels [32], Superpixels Extracted via Energy-Driven Sampling [34] and Extended Topology Preserving Segmentation [35] are based on an energy optimization process in which the image is partitioned into a regular grid as initial superpixel segmentation and pixels are exchanged between neighboring superpixels to optimize a formulated energy [30]. Entropy Rate Superpixels [31] is a graph-based algorithm that treats the image as undirected graph and partition this graph based on edge-weights which are often computed as color differences or similarities [30]. Eikonal Region Growing Clustering [33] represents superpixels as evolving contours starting from initial seed pixels [30]. SLIC [12] is a clustering based algorithm inspired by k-means initialized by seed pixels and combining color information and spatial information [30].

However, these algorithms still generate some mixed superpixels which stretch across different objects. The reasons for the mixed superpixels are various. For example, in SLIC algorithm, if the initial cluster centers locate near the boundary, the superpixels related to these cluster centers will include nearby pixels from different objects because the spatial distance and spectral distance are both considered in the clustering process. The classification on mixed superpixels certainly generated incorrect result for some pixels because only one label is assigned to all pixels in the mixed superpixel.

In this paper, a purification algorithm is proposed to purify SLIC superpixels for reducing misclassifications caused by mixed superpixels in the superpixel-based classification. In addition, an efficient scheme to apply the purified SLIC superpixels in the classification is proposed. The mixed SLIC superpixels are divided into small pure superpixels by applying the proposed purification algorithm, which is based on the color quantization [36]. The result of color quantization provides the information for judging whether the SLIC superpixel should be purified and locating new cluster centers for the purifying. With the proposed purification algorithm, the generated superpixels are purer and can preserve more accurate boundary information. The purified SLIC superpixels and original SLIC superpixels generated on images from Berkeley Segmentation Dataset 500 (BSDS500) were compared in three different evaluation metrics to validate the effectiveness of the proposed purification algorithm. The experimental results show that the purified SLIC superpixels perform better than original SLIC superpixels for all evaluation metrics. As for the classification scheme, purifying all superpixels in a large high spatial resolution remote sensing image is unnecessary and some unnecessary purifying may cause misclassification. In the proposed classification scheme, only superpixels which are in the group of edge superpixels are considered for purifying. In this way, most superpixels that do not need to be purified can be excluded for purifying process. To demonstrate that the purified SLIC superpixels can provide a more accurate basis for subsequent process, the proposed classification scheme was compared with the classification based on the SLIC superpixels in classification accuracy. In addition, the classification accuracy of the proposed scheme and the scheme which purifies all SLIC superpixels were compared to show that purifying all SLIC superpixels is unnecessary and may cause the loss of classification accuracy. The experimental results show that the classification based on purified SLIC superpixels in different scales can get higher accuracy than classification based on original SLIC superpixels and the scheme of purifying all SLIC superpixels indeed causes the loss of classification accuracy.

2. Background and Methods

2.1. Simple Linear Iterative Clustering (SLIC) Superpixels

Among various algorithms proposed to generate superpixels, SLIC has been proved to be on the list of algorithms showing superior performance [30] regarding metrics such as boundary recall, under-segmentation error, and explained variation. It also shows advantages in simplicity, adherence to boundaries, computational speed and memory efficiency. SLIC is a clustering based algorithm inspired by methods such as k-means initialized by seeds pixels, and using spectral information and spatial information. The parameters that need to be assigned in SLIC are k, which indicates the number of superpixels to be generated, and a parameter m, which controls the compactness of the superpixels [26]. With the parameter k, the average size of superpixels $S\times S$ can be calculated. The parameter m can control the spatial proximity that is considered in the calculation of distance. The commonly used value 10 is assigned to m in this study. In the first step of SLIC, k initial clusters centering on a square grid with the size of $S\times S$ are generated. Then, cluster centers are perturbed in a $3\times 3$ neighborhood to the lowest gradient position to avoid placing the seed on an edge or a noisy pixel [12]. After the locations of all initial cluster centers are confirmed, an iterative procedure is applied to update cluster centers. The iterative procedure can be divided into three steps:

(1)

For each cluster center, calculate the distance D between the cluster center and pixels within the $2S\times 2S$ search space centered by the cluster center and assign each pixel to the nearest cluster center.

(2)

Update new cluster centers by calculating the mean vector of all the pixels belonging to the superpixel.

(3)

Calculate the residual error E, which is the distance between previous center locations and updated center locations.

The iterative procedure will stop when the residual error converges or the number of iterations reaches the threshold. The threshold of iterations used in this paper is 10 because 10 iterations is sufficient for most images according to Achanta et al. [12]. Finally, disjoint pixels are reassigned to nearby superpixels to enforce connectivity. The calculation of the distance D is as follows:

$$D=\sqrt{{\left(\frac{d_c}{m}\right)}^2+{\left(\frac{d_s}{S}\right)}^2}$$

(1)

$$d_c=\sqrt{{(l_i-l_j)}^2+{(a_i-a_j)}^2+{(b_i-b_j)}^2}$$

(2)

$$d_s=\sqrt{{(x_i-x_j)}^2+{(y_i-y_j)}^2}$$

(3)

where $d_c$ and $d_s$ represent the color and spatial distance between pixels $I(x_i,y_i,l_i,a_i,b_i)$ and $I(x_j,y_j,l_j,a_j,b_j)$, $(x_i,y_i)$ is the coordinate of pixel i and $(l_i,a_i,b_i)$ refers to the color components of pixel i in CIELab color space [37].

2.2. SLIC Superpixels Purification Algorithm Based on Color Quantization

After SLIC, there are some mixed superpixels which contain pixels from different classes. In Figure 1, the superpixels circled in red are mixed superpixels. These mixed superpixels are mainly caused by the improper locations of initial cluster centers in SLIC. If the initial cluster center is near the boundary, a mixed superpixel may be generated because the spatial proximity is taken into consideration in the calculation of distance D. To avoid the error caused by these mixed superpixels in subsequent process, a SLIC superpixels purification algorithm based on color quantization is proposed in this section. It aims at dividing mixed superpixels into pure small superpixels. The procedure of the proposed superpixels purification algorithm is shown in Figure 2.

Color quantization is a process that tries to reduce the number of colors in a given image to a certain limited number given by the external factors, while keeping a visual essence of that image intact [38,39]. I is usually used to satisfy display capabilities of certain devices, to reduce a raw size of the image. The color counts in a source image are usually much higher than those allowed, so the algorithm has to discard the colors carefully. As the Wu’s Color Quantizer [36] can provide very good picture quality unreachable by other algorithms [39], it is selected to quantize the colors in the mixed superpixels. Based on the observation that most mixed superpixels contains two different classes, the number of colors assigned for Wu’s Color Quantizer is 2. It should be noted that in the process of cutting a color cube into two sub-cubes, if the superpixel is pure, the cutting will be ignored. Therefore, in the quantization, if the superpixel is very pure, the number of colors in the output will be changed to be 1 automatically. After quantization, if the superpixel is not pure, the outputs of Wu’s Color Quantizer are two extracted colors and a map with the color labels.

The purifying process is based on the outputs of the color quantization which can detect different colors in the superpixels. There are two conditions in which the superpixels will not be considered as mixed: (1) the superpixel is very pure so that the output of the color quantization algorithm is a single color; and (2) the distance between the two extracted colors is less than the threshold T. These two conditions are shown in two diamond boxes in Figure 2. Assuming that two colors extracted by the color quantization algorithm are $C_1=(l_1,a_1,b_1)$ and $C_2=(l_2,a_2,b_2)$ with the color components in CIELab color space, instead of calculating the Euclidean distance, the CIELab color distance between $C_1$ and $C_2$ is calculated with CIEDE2000 Color-Difference Formula [40], which is more precise.

If a superpixel is labeled as mixed, it will be processed by an iterative cluster procedure which is similar to SLIC. First, two cluster centers are generated by calculating the mean vector of all the pixels belonging to the $C_1$ and $C_2$ according to the labeled map generated from color quantization. Then, the distance D between each cluster center and pixels in the superpixels are calculated. According to the calculated distance, each pixel is assigned to the nearest cluster center. Next, new cluster centers are updated by calculating the mean vector of all the pixels belonging to the same cluster center. As same as SLIC, the iterative procedure will stop when the residual error converges or the number of iterations reaches the threshold 10. After enforcing connectivity, more than one new superpixels will be generated. These small superpixels are purer than the original mixed superpixel.

2.3. High Spatial Resolution Remote Sensing Image Classification Based on Purified SLIC Superpixels

The SLIC superpixels have been used for the classification of high spatial resolution remote sensing image in [1]. However, if the generated superpixels are not pure, the classification result of these superpixels must be incorrect. Therefore, the proposed purified SLIC is applied to the classification for getting better result. As the size of the high spatial resolution remote sensing image is always large, the number of generated SLIC superpixels will also be large. Some interior superpixels in the ground objects do not need to be purified. Therefore, to avoid unnecessary purifying, an efficient method which can detect interior superpixels is needed. Based on the observation that most mixed superpixels locate near the boundaries separating different classes, a mixed superpixels detection scheme utilizing the information of classification result is applied in this section. At first, a classification based on the SLIC superpixels will be executed. In the generated classification map, superpixels are divided into two groups, edge superpixel and interior superpixel (as shown in Figure 3), according to superpixel’s class label and its surrounding neighbor superpixels’ class labels. For the superpixel $S_i$ with the class label $L_i$, assuming that its neighbor superpixels belong to a set $S_N=\{{S_N}_1,{S_N}_2,\cdots ,{S_N}_{\mathrm{j}},\cdots \}$, the group label $G\left(S_i\right)$ is defined as follows:

$$G\left(S_i\right)=\{\begin{array}{cc}\mathbf{edge}\mathbf{superpixel}\hfill & \hfill \exists {S_N}_{\mathrm{j}}\in S_N|{L_N}_{\mathrm{j}}\ne L_i\\ \mathbf{interior}\mathbf{superpixel}\hfill & \hfill \forall {S_N}_{\mathrm{j}}\in S_N|{L_N}_{\mathrm{j}}=L_i\end{array}$$

(4)

Then, all edge superpixels are the detected candidate mixed superpixels. The purifying process will be executed in these edge superpixels. After purifying, all superpixels generated from edge superpixels will be reclassified. The final classification result is the union result of the interior superpixels and the purified edge superpixels. The outline of the proposed classification scheme is illustrated in Figure 4.

3. Datasets and Experimental Results

3.1. Datasets

The Berkeley Segmentation Dataset 500 [41,42] was the first to be used for superpixel algorithm evaluation. It contains 500 images and provides at least five high-quality ground truth segmentations per image. The images represent simple outdoor scenes, showing landscape, buildings, animals and humans, where foreground and background are usually easily identified. We chose three images with the size of $481\times 321$ from the BSDS500 to validate the advantages of the proposed purification algorithm. As each image has at least five high-quality ground truth segmentations, we evaluated the generated superpixels on all ground truth segmentations and calculated the mean value for each metric. The image and one of the ground truth segmentations are shown in Figure 5.

In terms of high spatial resolution remote sensing image, we chose an image with the resolution of 0.5 m obtained from WorldView-2 satellite to validate the performance of the Purified SLIC superpixels. The selected image with the size of $5000\times 4000$ pixels covers the residential and forest area of the city Oromocto in Canada. The image contains four bands: red, green, blue and NIR band. The reference map was generated from an object based classification result with seven classes: tree, grass, road, light roof, dark roof, water and bare soil. Then, the label of misclassified regions were modified manually by an independent experienced researcher. The image and the reference map are shown in Figure 6.

3.2. Segmentation Quality Comparison between Purified SLIC Superpixels and Original SLIC Superpixels

We used three metrics [30], namely Boundary Recall (Rec) [43], Under-segmentation Error (UE) [12,15,44], and Explained Variation (EV) [45], to evaluate the performance of the purified SLIC superpixels and original SLIC superpixels. In the following, let $S={\left\{S_j\right\}}_{j=1}^K$ and $G=\left\{G_i\right\}$ be partitions of the same image and $I:x_n\to I\left(x_n\right)$, $1\le n\le N$ be an image with $x_n$ being the pixel location and $I\left(x_n\right)$ the corresponding intensity or color. The Rec is a commonly used metric to asses boundary adherence given ground truth. Let $FN(G,S)$ and $TP(G,S)$ be the number of false negative and true positive boundary pixels in S with respect to G. Then, Rec is defined as:

$$\mathrm{Rec}(G,S)=\frac{TP(G,S)}{TP(G,S)+FN(G,S)}$$

(5)

UE measures the “leakage” of superpixels with respect to G and, therefore, implicitly also measures boundary adherence. Here, “leakage” refers to the overlap of superpixels with multiple, nearby ground truth segments. The formulation is computed as:

$$\mathrm{UE}(G,S)=\frac{1}{N}\sum _{G_i}\sum _{G_i\cap S_j\ne \varnothing }min\{|S_j\cap G_i|,|S_j-G_i|\}$$

(6)

EV quantifies the quality of a superpixel segmentation without relying on ground truth. As image boundaries tend to exhibit strong change in color and structure, EV assesses boundary adherence independent of human annotations. It is defined as:

$$\mathrm{EV}(G,S)=\frac{{\sum }_{S_j}\left|S_j\right|{(\mu \left(S_j\right)-\mu \left(I\right))}^2}{{\sum }_{x_n}{(I\left(x_n\right)-\mu \left(I\right))}^2}$$

(7)

where $\mu \left(S_j\right)$ and $\mu \left(I\right)$ are the mean color of superpixel $S_j$ and the image I, respectively.

It should be noted that, for generated superpixels, higher Rec and EV are better while lower UE is better. With these metrics, we compared the segmentations of the three initial superpixel sizes $10\times 10$, $15\times 15$ and $20\times 20$ generated from purified SLIC and original SLIC in the three images selected from BSDS500. The calculated metrics are shown in

Table 1. Evaluation metrics for superpixels.

	Image 24,063			Image 29,030			Image 201,080
	Rec	UE	EV	Rec	UE	EV	Rec	UE	EV
SLIC$10\times 10$	0.8347	0.0466	0.9736	0.8261	0.0600	0.9170	0.8753	0.0514	0.9268
Purified SLIC$10\times 10$	0.8727	0.0360	0.9865	0.8622	0.0434	0.9453	0.9150	0.0328	0.9583
SLIC$15\times 15$	0.7743	0.0641	0.9620	0.7840	0.0847	0.8821	0.8391	0.0728	0.9049
Purified SLIC$15\times 15$	0.8338	0.0435	0.9834	0.8208	0.0579	0.9342	0.9030	0.0406	0.9507
SLIC$20\times 20$	0.7630	0.0776	0.9540	0.5790	0.1051	0.8737	0.7328	0.0937	0.8859
Purified SLIC$20\times 20$	0.8202	0.0527	0.9787	0.6523	0.0706	0.9308	0.7883	0.0479	0.9422

. The segmentation maps with the initial superpixel size $20\times 20$ for three images are shown in Figure 7. As can be observed, the purified SLIC superpixels outperformed the SLIC pixels for all images in terms of all metrics. Moreover, from segmentation maps in Figure 7, it can be seen that obvious mixed SLIC superpixels are divided into pure superpixels by the proposed purification algorithm.

3.3. Comparison of Remote Sensing Image Classification Results Generated from Purified SLIC Superpixels and Original SLIC Superpixels

We used three metrics, namely overall accuracy (OA), average accuracy (AA) and kappa coefficient, to evaluate the performance of classification based on the purified SLIC and original SLIC superpixels in the image of the city Oromocto in Canada. As in [1], we chose the Random Forest (RF) classifier [46] to do the classification. RF classifier builds a set of trees that are created by selecting a subset of training samples through a bagging approach, while the remaining samples are used for an internal cross-validation to estimate how well the RF model performs [47]. The number of trees in the RF that should be assigned by users was set as 500, which had been shown to be sufficient in order to stabilize the errors before reaching this number of classification trees [47]. The object attributes used for classification included: (1) spectral attributes of mean value and standard deviation of each band for each superpixel, brightness, and NDVI; and (2) gray level co-occurrence matrix (GLCM) texture attributes [48] of GLCM standard deviation, GLCM homogeneity and GLCM correlation. The shape attributes were excluded because the superpixel is too small to describe the real shape information of ground object and most superpixels have similar sizes and shapes. As the test image contains four bands, the number of calculated object attributes for each superpixel is 13. In the stage of training, to guarantee the purity of sample superpixels, we filtered the SLIC superpixels with a rule that there must be a class occupying 90% pixels in the superpixel. Then, 50% of the filtered SLIC superpixels were randomly selected as samples and the remaining SLIC superpixels were used to evaluate the performance of the classification. The number of generated superpixels and selected samples with three initial superpixel sizes of $10\times 10$, $15\times 15$ and $20\times 20$ are listed in

Table 2. Number of SLIC superpixels and selected samples for three scales.

	$10\times 10$	$15\times 15$	$20\times 20$
Tree	40,621	15,555	7457
Light Roof	564	189	90
Road	8394	3271	1614
Grass	16,651	6108	2879
Dark Roof	2069	715	374
Bare Soil	4212	1483	677
Water	4654	1919	998
Total	77,165	29,240	14,089
SLIC superpixels	205,700	87,442	46,778

. The color distance threshold T is 6, which is used in the purifying stage to judge whether the edge superpixels should be purified or not. It should be noted that the SLIC superpixels and purified SLIC superpixels were both classified with the same classification model, which was trained from filtered SLIC superpixels. All tests were conducted on a computer with an Intel Core i5-7300HQ CPU (2.50 GHz) processor with 8 GB RAM, using a 64-bit Windows 10 operating system.

With three evaluation metrics mentioned above, we compared the classification results based on purified SLIC and original SLIC superpixels with different initial superpixel sizes. In addition, to demonstrate that purifying all superpixels is unnecessary, the classification based on the scheme purifying all SLIC superpixels was also investigated. The SLIC superpixels and purified SLIC superpixels generated in a subset image of $512\times 512$ pixels cut from the test image are shown in Figure 8. The average classification accuracy in ten runs of classification for the whole image are listed in the

Table 3. Average classification accuracy in ten runs of classification.

	OA(%)	AA(%)	Kappa
SLIC$10\times 10$	77.74	67.97	0.697
Purified SLIC$10\times 10$	79.68	71.29	0.725
All Purified SLIC$10\times 10$	79.51	71.26	0.722
SLIC$15\times 15$	75.99	65.15	0.674
Purified SLIC$15\times 15$	79.41	70.83	0.722
All Purified SLIC$15\times 15$	79.20	70.80	0.720
SLIC$20\times 20$	74.12	62.80	0.648
Purified SLIC$20\times 20$	78.80	70.46	0.714
All Purified SLIC$20\times 20$	78.62	70.42	0.712

. As some superpixels are not pure, the classification accuracy was calculated based on individual pixels within superpixels. The classification maps for one run are shown in Figure 9. As can be observed, the classification based on purified SLIC superpixels outperformed the classification based on SLIC superpixels in all scales. The overall accuracy increases are, respectively, 1.94%, 3.42% and 4.68% for scales $10\times 10$ to $20\times 20$. To better show the performance of the proposed method, we selected some small plots in the image to demonstrate how the proposed purification algorithm optimizes the classification (as shown in Figure 10). For each plot, the SLIC superpixel circled in red is mixed and the related classification is incorrect while the generated purified SLIC superpixel is pure and the related classification is correct. In addition, the accuracy of the scheme purifying all superpixels is a little lower than the proposed scheme purifying edge superpixels because unnecessary purifying for some SLIC superpixels leads to misclassification.

4. Discussion

The comparison of classification accuracy shows that the classification based on purified SLIC superpixels outperformed the classification based on SLIC superpixels. However, from the outline of the proposed classification, it is easy to find that the proposed classification has three additional steps compared to the classification based on SLIC superpixels. It means that the proposed classification will take more time than the classification based on SLIC superpixels. The average time for each step in ten runs of classifications are tabulated in

Table 4. Average time cost (s) for each step in ten runs of classification.

	SLIC $10\times 10$	Purified SLIC $10\times 10$	SLIC $15\times 15$	Purified SLIC $15\times 15$	SLIC $20\times 20$	Purified SLIC $20\times 20$
SLIC	27	27	23	23	22	22
Feature Extraction	5230	5230	2177	2177	1174	1174
RF Training	212	212	74	74	33	33
Classification	30	30	11	11	5	5
Purifying	–	110	–	93	–	92
Feature Extraction	–	2584	–	2155	–	1752
Classification	–	17	–	12	–	8
Total	5499	8210	2285	4545	1234	3086

. With the increase of initial SLIC superpixel size, the time decreases because less SLIC superpixels are generated. We can find that the feature extraction is the most time-consuming step in the classification, occupying more than 90% of the time in classification. Compared with the classification based on SLIC superpixels, the proposed classification generates a lot of new small superpixels so that the features of these new superpixels need to be calculated. Thus, calculating features for new superpixels also costs most time among the three additional steps in the proposed classification. Moreover, the purifying step takes more than four times that of SLIC. As for the total time, the proposed classification costs 2711 s, 2260 s and 1852 s more than the classification based on SLIC superpixels for scales $10\times 10$ to $20\times 20$, respectively. Despite the increased time cost is significant, the improvement in the classification accuracy is encouraging.

There are two factors that can influence the running time of the classification based on purified SLIC superpixels. The first factor is the complexity of the test image. If the test image contains less boundary between different classes, the number of detected edge superpixels will decrease, thus reducing time cost of subsequent procedure. The second factor is the color distance threshold T used in purifying step. The color distance threshold T determines whether the edge superpixel should be purified or not according to the output of color quantization. If the T is too small, the number of edge superpixels need to be purified is large so that the number of new superpixels generated in purifying is large. Even some superpixels that are interpreted as pure superpixels by human analysis will be purified. The time cost for calculating features of new superpixels will be high. On the contrary, if T is too large, the number of edge superpixels that need to be purified is small so that the number of new superpixels generated in purifying is small. The time cost for calculating features of new superpixels will be low. However, some mixed edge superpixels may not be purified. Therefore, the T not only influences the time cost of the proposed classification, but also affects the accuracy of the classification. To investigate the influence of the color distance threshold T for the proposed classification, we ran the proposed classification with different T, from 4 to 20 with an interval of 2. The classification accuracy and time cost for each T are shown in Figure 11 and Figure 12, respectively. In Figure 11, the OA of the proposed classification continually declines with the increase of T for three scales. However, the OA of the proposed classification is obviously higher than the classification based on SLIC superpixels even with a T of 20, which is large enough for judging that two colors are similar. It means that the color difference in some mixed SLIC superpixels is very explicit. As for the time cost, the line chart in Figure 12 shows that the time cost reduces with the growing of T. When the T reaches 20, the difference of the time cost between the proposed classification and the classification based on SLIC superpixels is below 500 s. As can be concluded, the choice of T is a balance of performance and efficiency, and choosing a T which can improve the performance of subsequent processes is not difficult for users.

Some limitations also exist in the proposed purification algorithm. For example, we fix the number of colors at two for the color quantization process because most mixed superpixels only contain pixels from two different classes. However, a few mixed superpixels containing pixels from more than two different classes are still generated. For these mixed superpixels, the results of color quantization are not accurate, thus the purifying cannot divide the superpixel accurately. The current solution for this problem is to run the purifying process multiple times. For each mixed superpixel, the purifying will not stop unless all generated new superpixels cannot be divided. In this way, the generated new superpixels will be pure no matter how many classes are contained in the original superpixel. Another limitation is that a few superpixels that are identified as pure by human eyes may also be purified. For example, as shown in Figure 7f, some pure grass superpixels on the bottom of the picture are purified to be small superpixels. The reason is that the color variation for these grass superpixels are higher than the assigned color distance threshold. Although most unnecessary purifying does not negatively affect the accuracy of subsequent process, the extra time cost for the purifying will lower the efficiency. Therefore, developing methods to avoid the unnecessary purifying is a good direction for further research. In addition, although the proposed purification algorithm is applied to SLIC superpixels in this paper, it can also be used to purify mixed superpixels generated by other superpixel algorithms.

5. Conclusions

In this paper, a SLIC superpixels purification algorithm based on color quantization is proposed to improve the accuracy of superpixel-based high spatial resolution remote sensing image classification. The novelty of this study consists of two factors: (1) using the quantized color information in the SLIC superpixel to purify mixed superpixels; and (2) applying a classification scheme to avoid purifying superpixels that do not need to be purified. The first factor successfully reduces the misclassification caused by mixed SLIC superpixels. The second factor reduces the unnecessary purifying process and reserves pure superpixels in the original scale. The experiments on images from BSDS500 demonstrate that the purified SLIC superpixels are better than SLIC superpixels in all three metrics for evaluating superpixels. The experiments on the high resolution remote sensing image demonstrate that, compared with the classification based on SLIC superpixels, the classification based on purified SLIC superpixels can achieve the accuracy increase of 1.94%, 3.42% and 4.68% at scale $10\times 10$, $15\times 15$, and $20\times 20$, respectively.