Multiclass Contour-Preserving Classification with Support Vector Machine (SVM)

2.1 SVM

SVM [3] is a set of related supervised learning methods that analyzes data and recognizes patterns used for classification and regression analysis. A standard SVM predicts which of two possible classes an input is a member of. A prediction is done by constructing a hyperplane or a set of hyperplanes in a high- or infinite-dimensional space that will maximize a margin between both classes. In a data set where the data is linearly separable, hard-margin linear SVM will select instances, which lie along the supporting hyperplanes (the hyperplanes parallel to the dividing hyperplane at the edges of the margin), as support vectors. If the data set is not linearly separable, soft-margin linear SVM can widen the margin and produce a greater error for the data set, but improve generalization and/or find a linear separation of data that is not linearly separable. Besides, the data set that is not linearly separable in its original feature space can be transformed into a higher-dimensional space before the non-linear SVM divides the data via a kernel function. After data division is completed, the higher-dimensional space is transformed back to its original feature space. The performance of SVM may be considered regarding accuracy as well as computational complexity. The accuracy of the classification depends on the trade-off between a high-complexity model, which may over-fit the data, and a large margin, which will incorrectly classify some of the data set in the interest of better generalization. In general, better generalization error of the classifier can be achieved when the margin is larger. More specifically, a good separation is achieved by the hyperplane that has the largest distance to the neighboring data points (vectors) of both classes. The computational complexity of the classification focuses on the training time, which depends on the quadratic function for linear SVM, and the selected kernel function for non-linear SVM. For a testing time, the computational complexity is linear in the number of support vectors. Fewer support vectors lead to faster classification. Figure 2 illustrates support vectors and their hyperplane. An LIBSVM library [2] has been implemented to support multiclass data.

Figure 2:

Support Vectors and their Hyperplane.

2.2 MCOV

MCOV [6] is a technique that can narrow the space between consecutive classes of data to preserve the concave surface (curves inward) and the convex surface (bulges outward) of the data more accurately. The technique has been improved from Ref. [8] to support multiclass data and confirmed in Ref. [4] to improve the level of classification accuracy on FFNN. The technique synthesizes two types of new instances, FMCOV and AMCOV, from original instances in the middle of the decision boundary between consecutive classes of data, using Euclidean distance function. The anatomy of both FMCOV and AMCOV is shown below.

1. FMCOV is a synthesized vector that is used to declare the decision boundary of the territory of an instance of one class, let presume an instance i of class A [denoted by A_i], against an instance of any other class, let presume an instance j of class X [denoted by X_j], that has the smallest Euclidean distance to A_i. X_j is designated as a paired vector of A_i [denoted by ϕ(i)]. An FMCOV of A_i [denoted by o(i)] is placed at the boundary of A_i’s territory in the direction of X_j.

2. AMCOV is a synthesized vector that is used to declare the decision boundary of a paired vector of an instance, let presume a paired vector of instance i of class A [denoted by ϕ(i)], against that instance i of class A [denoted by A_i]. An AMCOV of ϕ(i) [denoted by o′(i)] is placed at the boundary of ϕ(i)’s territory; called counter boundary, in the direction of A_i.

The Multiclass Outpost Vector Generation Algorithm is presented in Algorithm 1. Figure 3 illustrates the concept of the three-class outpost vector [6] to help understand the anatomies of FMCOV and AMCOV. In Figure 3, there are three classes of data, which are designated as class A, B, and C. The top left and bottom center circles are the territory of class A. The top right and bottom left circles are the territory of class B. The bottom right circle is the territory of class C. To find the territory of each instance, each instance is modeled to span its territory as a circle (sphere in a case of three-dimensional space or hypersphere in a case of higher-dimensional space) until the territories collide with any other. The territory of instance k of class A, denoted by A_k, is found by locating an instance not in class A that is nearest to A_k. In this case, B*(A_k) of class B is nearest to A_k and referred to as A_k’s pair. Then, the territory of A_k is declared at halfway between A_k and B*(A_k). Consequently, the radius of A_k’s territory is set at half of the distance between A_k and B*(A_k). This option is to guarantee that if B*(A_k) sets its territory using the same radius, then the distance from the hyperplane to either A_k or B*(A_k) will be at the maximum. A_k then places its FMCOV against B*(A_k) at the decision boundary of A_k. The territories of B*(A_k) of class B, C*(B*(A_k)) of class C, A_j of class A, and B*(A_j) of class B are also found by the same method done with A_k of class A. After that, AMCOVs will then be generated from all instances as well. The AMCOV of B*(A_k) of class B against A_k of class A is placed at the counter boundary of B*(A_k) against A_k. The AMCOV of C*(B*(A_k)) of class C against B*(A_k) of class B is placed at the counter boundary of C*(B*(A_k)) against B*(A_k). The AMCOV of A_j of class A against C*(B*(A_k)) of class C is placed at the counter boundary of A_j against C*(B*(A_k)). The AMCOV of B*(A_j) of class B against A_j of class A is placed at the counter boundary of B*(A_j) against A_j.

Figure 3:

Instances (Big Rectangles), FMCOVs (Small Rectangles), AMCOVs (Small Triangles), Instances’ Boundary (Solid Circles), Instances’ Counter Boundary (Dotted Circles) in a Two-Dimensional Three-Class Data Set.

MCOV has been augmented in Ref. [7] to solve a space deficiency of an FMCOV generator and an AMCOV generator by a postprocessor that can remove a set of MCOVs that is not located close to the decision boundary between consecutive classes of data.

4.1 Machine Learning

The experiments were carried out using MATLAB R2015a and LIBSVM v3.20 [2]. The following parameters were used:

• SVM type=C-SVC (multiclass classification)

• Kernel function=radial basis function

• Degree in Kernel function=3

• Gamma in Kernel function=1

• Coefficient 0 in Kernel function=0

• Cost of C-SVC=1

• Weight of C-SVC=1

• Cache memory size=100 MB

• Tolerance of termination criterion=0.001

• Shrinking heuristics=1

• Probability estimates=0

4.2 Data Sets

Three groups of data sets were used in the experiments:

1. Six non-overlapping four-class synthetic data sets

2. Eight highly overlapping two-class synthetic data sets from the ELENA project [9]

3. Six multiclass real-world data sets from the UCI machine learning repository [1]

Table 1 presents the characteristics of six non-overlapping four-class synthetic data sets. These data sets were used to demonstrate how MCOVs could narrow the space between consecutive classes of multiclass data. They were generated by the same algorithm, but having different population sizes. There were four classes of data, which are designated as red, green, blue, and magenta points in a two-dimensional Euclidean space determined by two coordinates. The data sets for training consisted of 100, 200, 400, 800, 1600, and 3200 instances per class, which constituted 400, 800, 1600, 3200, 6400, and 12,800 instances in six data sets, respectively. The data set for training having 3200 instances per class is depicted in Figure 4A. Figure 4B, C, and D depict the data set for training having 3200 instances per class with FMCOVs, AMCOVs, and both FMCOVs and AMCOVs, respectively.

Figure 4:

(A) A Four-Class Synthetic Data Set with 3200 Samples per Class with (B) FMCOVs, (C) AMCOVs, and (D) Both FMCOVs and AMCOVs [9].

Table 2 presents the characteristics of eight highly overlapping two-class synthetic data sets from the ELENA project [9]. There were “Clouds,” “Gaussian 2D,” “Gaussian 3D,” “Gaussian 4D,” “Gaussian 5D,” “Gaussian 6D,” “Gaussian 7D,” and “Gaussian 8D” data sets. These data sets were used to evaluate how MCOVs could improve the level of SVM classification accuracy on two-class data sets having a heavy intersection of the class distributions, a high degree of non-linearity of the class boundaries, and various dimensions of the vectors. The data sets for training and testing of each data set were generated by four-fold cross validation. The “Clouds” data set for training, FMCOVs, AMCOVs, and both FMCOVs and AMCOVs are depicted in Figure 5A, B, C, and D, respectively. The “Gaussian 2D” data set for training, FMCOVs, AMCOVs, and both FMCOVs and AMCOVs are depicted in Figure 6A, B, C, and D, respectively.

Figure 5:

(A) “Clouds” Highly Overlapping Two-Class Synthetic Data Set having 3750 Samples. (B) FMCOVs. (C) AMCOVs. (D) Both FMCOVs and AMCOVs. [9].

Figure 6:

(A) “Gaussian 2D” Highly Overlapping Two-Class Synthetic Data Set having 3750 Samples. (B) FMCOVs. (C) AMCOVs. (D) Both FMCOVs and AMCOVs. [9].

Table 3 presents the characteristics of six multiclass real-world data sets from the UCI machine learning repository [1]. There were “Adult Income,” “Statlog (Landsat Satellite),” “Statlog (Shuttle Landing Control),” “Forest Cover Level,” “Pen-Based Recognition of Handwritten Digits,” and “Optical Recognition of Handwritten Digits” data sets. These data sets were used to evaluate how MCOVs could improve the level of SVM classification accuracy on multiclass real-world data sets having various data complexities, including the number of classes and the number of features or dimensions. The data set for training and the data set for testing of each data set were provided.

The κ parameter [6] used in FMCOV generator and AMCOV generator on all final training sets was set equally to 5%. Table 4 summarizes the components of the final training sets aforementioned.

4.3 Experimental Results

Based on the methodology presented in Section 3 with SVM configuration presented in Section 4.1 on the data sets presented in Section 4.2, the following experimental results were produced.

Table 5 presents the SVM misclassification rates on the six non-overlapping four-class synthetic data sets. For this group of data sets

• The final training sets having raw data and FMCOVs (S_Raw+FM) and the final training sets having raw data and both FMCOVs and AMCOVs (S_Raw+M) yield the lowest and the second-lowest SVM misclassification rates, respectively.

• The final training sets having raw data and FMCOVs (S_Raw+FM), the final training sets having raw data and AMCOVs (S_Raw+AM), and the final training sets having raw data and both FMCOVs and AMCOVs (S_Raw+M) yield a higher level of SVM classification accuracy than that of the final training sets having raw data only (S_Raw) and the final training sets having only some types of MCOVs (S_FM, S_AM, S_M).

Table 6 presents the SVM misclassification rates on the eight highly overlapping two-class synthetic data sets. For this group of data sets

• Both the final training sets having raw data only (S_Raw) and the final training sets having raw data and FMCOVs (S_Raw+FM) yield the lowest SVM misclassification rate.

• An inclusion of AMCOVs on the final training sets (S_AM, S_Raw+AM, or S_M) can adversely affect the level of SVM classification accuracy.

Table 7 presents the SVM misclassification rates on the six multiclass real-world data sets. For this group of data sets

• The final training sets having raw data and FMCOVs (S_Raw+FM) yield the lowest SVM misclassification rate.

• The experimental results from “Adult Income,” “Statlog (Shuttle Landing Control),” and “Optical-Based Recognition of Handwritten Digits” data sets show that the final training sets having raw data and FMCOVs (S_Raw+FM) can significantly improve the level of SVM classification accuracy.

• The experimental results from “Forest Cover Level” and “Pen-Based Recognition of Handwritten Digits” data sets show that the final training sets having raw data and FMCOVs (S_Raw+FM) can slightly improve the level of SVM classification accuracy.

• The experimental results from “Statlog (Landsat Satellite)” data set show that the final training sets having raw data and both FMCOVs and AMCOVs (S_Raw+M) can slightly improve the level of SVM classification accuracy.

The experiment results show that an inclusion of FMCOVs on the final training sets having raw data (S_Raw+FM) can improve the level of SVM classification accuracy. Hence, it can be concluded that the MCOV is applicable with both FFNN and SVM.