Feature Selection Using Harmony Search for Script Identification from Handwritten Document Images

1 Introduction

Many strategies have been exploited for the task of feature selection (FS), in an effort to identify more compact and better-quality feature subsets. The development of nature-inspired stochastic search techniques allows multiple good-quality feature subsets to be discovered without resorting to exhaustive search. Selecting a subset of features from the entire feature set is often beneficial because it lessens the computation time along with helping achieve better classification accuracy. At a higher dimension, it becomes very difficult to visualize the data and there may be some non-contributing features that may even mislead the classifier’s decision [31]. Identification of those features at higher dimension, though a challenging task, motivates the researchers to work in this domain. For a large feature dimensionality, it is almost impossible to check the accuracies considering all possible feature subsets. Thus, one may apply the hill climbing (HC) method by successively adding/removing feature(s) into/from the original feature set. However, the HC method gets stuck at local minima. Thus, the need to explore the search space using some heuristic search techniques will be of utmost importance. Practical problems that arise during analyzing the real-world data are often related to the number of features (so called curse of dimensionality). The inability of the classification tools to identify and extract patterns or rules easily may also be attributed to high interdependency among individual features, or the behavior of the combined features. Techniques such as text processing and classification can benefit greatly from FS, once the noisy, irrelevant, redundant, or misleading features are removed. Given a feature set with size n, there will be 2ⁿ possible feature subsets. The FS problem is to find a minimal feature subset while retaining a suitably high accuracy in classifying the data [25].

The FS approaches can generally be divided into two groups: filter and wrapper approaches. The filter approach operates independently of any learning algorithm. These methods rank the features by some criteria and neglect all features that do not achieve a sufficient score. Due to their computational efficiency, the filter methods are very popular for high-dimension data. Some popular filter methods are F-score criterion [7], mutual information [40], information gain [1], correlation [17], etc. On the other hand, the wrapper approaches involve a predetermined learning model while selecting features on measuring the parameters of the particular machine learning model [17]. Filter approaches are usually less computationally intensive than wrappers, but they produce a feature set that is biased to a specific type of predictive model. This unbiased tuning means a feature set from a filter is more general than the set from a wrapper, usually giving lower prediction performance than a wrapper. However, the feature set selected using the former method does not contain the assumptions of a prediction model, and so is more useful for exposing the relationships among the original features.

The rest of the paper is organized as follows: Section 2 presents a brief review of some of the previous approaches to FS, and Section 3 illustrates the motivation behind the proposed work. Section 4 describes the Modified log-Gabor filter-based feature extraction procedure for handwritten Indic script identification, whereas in Section 5, the basics of the harmony search (HS)-based feature subset selection process are discussed. Section 6 describes the performance of the proposed script identification system, and finally, Section 7 concludes the work.

2 Related Works

There has been an immense amount of work in the area of FS, which credits its importance. Sivagaminathan and Ramakrishnan [35] presented a hybrid method based on ant colony optimization (ACO) and artificial neural networks to address FS. The proposed hybrid model was demonstrated using datasets from the domain of medical diagnosis, yielding promising results. El Alami proposed a novel text categorization algorithm [9] called chaos genetic FS optimization in order to choose a subset of available features by eliminating unnecessary features to the classification task. The proposed algorithm selected the optimal subsets in both empirical and theoretical works in machine learning, and presented a general framework for text categorization. Experimental results showed that the proposed algorithm simplified the FS process effectively and obtained higher classification accuracy with only a subset of original features. Bermejo et al. [3] proposed a stochastic algorithm based on the greedy randomized adaptive search procedure (GRASP) meta-heuristic, with the main goal of speeding up the feature subset selection process, basically by reducing the number of wrapper evaluations to be carried out. GRASP is a multi-start constructive method that formulates a solution in its first stage, and then applies a modifier stage on that solution for an improved result. Several instances of the proposed GRASP method were experimentally tested and compared with state-of-the-art algorithms over 12 high-dimensional datasets. The statistical analysis of the results illustrated that the proposal was comparable in accuracy and cardinality of the selected subset to previous algorithms, but required significantly fewer evaluations. Forsati et al. [11] formulated the problem of FS as an optimization problem and introduced a novel FS procedure in order to achieve better classification results. Experiments over a standard benchmark demonstrated that applying bee colony optimization in the context of FS was a feasible approach. Das et al. [4] proposed a methodology where local regions of varying heights and widths were created dynamically on the pattern images. Genetic algorithm (GA) was then applied on these local regions to sample the optimal set of local regions containing best distinguishing information among the pattern classes. An optimal feature set was then extracted from those selected regions. Other popular optimization techniques like simulated annealing and HC had also been evaluated and, finally, a comparison with the proposed technique was done on a dataset of handwritten Bangla digits. Vieira et al. [39] developed modified binary particle swarm optimization (MBPSO) for feature subset selection. It optimized the kernel parameters of support vector machine (SVM) as well as controlled the premature convergence of the algorithm. Zeng et al. [41] introduced a novel hybrid FS method based on rough conditional mutual information and naive Bayesian classifier, which could also be used to filter the irrelevant and redundant features. Subsequently, to reduce the feature and improve classification accuracy, a wrapper approach based on naive Bayesian classifier was used to search the optimal feature subset in the space of a candidate feature subset that was selected by filter model. The results showed that the approach obtained not only high classification accuracy, but also selected the least number of features. Nazir et al. [27] proposed an efficient gender classification technique for real-world face images (labeled faces in the wild). This work extracted facial local features using local binary pattern (LBP), and then these features were fused with clothing features to enhance the classification accuracy. In the following step, PSO and GA were combined to select the most important features set that more clearly represented the gender, and thus the data dimensionality was reduced. Roy et al. [29] implemented artificial bee colony optimization to pin point the set of local regions offering the optimal discriminating feature set for handwritten numeral and character recognition. Initially, eight-directional gradient features were extracted from every region of different levels of partitions created using a center of gravity-based quad-tree partitioning approach. Then, using this approach, at each level, the sampling process was done based on SVM in every single region. De Stefano et al. [5] presented a novel GA-based FS algorithm in which feature subsets were evaluated by means of a specifically devised separability index. This index measured the statistical properties of the feature subset and did not depend on any specific classification scheme. The proposed index represented an extension of the Fisher linear discriminant method and used covariance matrices for estimating how class probability distributions were spread out in the considered N-dimensional feature space. A key property of the approach was that it does not require any a priori knowledge about the number of features to be used in the feature subset. Roy et al. [30] proposed a new FS methodology on the basis of features combined class separability power, using the framework of axiomatic fuzzy set theory, which provided the rules for logic operations needed to interpret the combinations of features from the fuzzy feature set. Based on these combinational rules, the class separability power of the combined features was determined, and subsequently the most powerful subset of the feature set was selected. Mohammadi and Abadeh [26] introduced a new feature-based blind steganalysis method for detecting stego images from the cover images in JPEG images using an FS technique based on the artificial bee colony optimization technique. Kashef and Nezamabadi-pour [18] presented a novel feature selection algorithm based on ACO, called advanced binary ACO. Features were treated as graph nodes to construct a graph model and were fully connected to each other. The performance of the proposed algorithm was compared with the performance of binary GA, binary PSO (BPSO), catfish BPSO, improved binary gravitational search algorithm, and some prominent ACO-based algorithms on the task of FS on 12 well-known UCI datasets. Simulation results verified that the algorithm provided a suitable feature subset with good classification accuracy using a smaller feature set than competing FS methods. Ahmad [2] used principal component analysis (PCA) for feature reduction and subset selection in which features were primarily projected into a principal space and then elected based on their eigenvalues, but the feature with the highest eigenvalue may not had the guarantee to provide optimal sensitivity for the classifier. To avoid this problem, two evolutionary optimization approaches like GA and PSO had been used to search the most discriminative subset of transformed features. Finally, it was observed that a feature subset selection based on PSO provided better performance as compared to GA. Singh et al. [32] proposed a PSO- and F-score-based FS algorithm for selecting the significant features that contributed to improve the classification accuracy. The performance of their proposed method was evaluated with various classifiers such as SVM, naive Bayes, k-NN, and decision tree. The experimental results illustrated that the proposed method outperformed some existing methods.

In particular, HS is a recently developed technique mimicking musicians’ experiences, which has been effectively utilized to cope with FS problems. Kumar et al. [22] presented a novel variance-based HS (VHS) algorithm for solving optimization problems. The problem of setting a constant parameter in the algorithm was removed, and the technique was named as explorative HS. The proposed algorithm was then employed for a data clustering problem on four real-life datasets chosen from the UCI machine learning repository. The results indicated that the VHS-based clustering outperformed the existing well-known clustering algorithms. Kumar et al. [21] introduced a parameter adaptive HS (PAHS) algorithm for solving optimization problems. The two important parameters of HS algorithm (HSA), namely harmony memory consideration rate (HMCR) and pitch adjusting rate (PAR), were allowed to dynamically change in their proposed algorithm in order to get the global optimal solution. Evaluation of the proposed algorithm for data clustering on five real-life UCI datasets showed better results than other algorithms. Kumar et al. [23] assumed the problem of automatic data clustering as the searching of optimal number of clusters in order to optimize the attained partitions. The PAHS algorithm (as described in Ref. [21]) was employed as an underlying optimization strategy. The data points of the different cluster centroids were assigned on the basis of newly developed weighted Euclidean distance. The algorithm was compared with four existing clustering techniques, and the same was further applied for automatic segmentation of grayscale and color images. Kumar et al. [20] proposed a novel automatic unsupervised FS method based on gravitational search algorithm, called AFSGSA (automatic FS using the gravitational search algorithm). The algorithm requires no prior knowledge of the data to be classified and the number of features to be selected. A novel fitness function was also included in order to make the search more efficient. Comparison of the performance of AFSGSA concluded the efficiency and efficacy of the proposed FS technique over other existing techniques for the same. In this paper, we present a wrapper FS method based on HS to discover a minimal feature subset from a problem domain while retaining a suitably high accuracy in representing the original data in the feature space. Applying this FS method, we have achieved some significant improvement in the recognition accuracy of the Modified log-Gabor filter-based handwritten Indic script identification technique.

3 Motivation

India is a multilingual country with 23 constitutionally recognized languages [38] written in 12 major scripts. Besides these, hundreds of other languages are used in India, each one with a number of dialects. The officially recognized languages are Hindi, Bengali, Punjabi, Marathi, Gujarati, Oriya, Sindhi, Assamese, Nepali, Urdu, Sanskrit, Tamil, Telugu, Kannada, Malayalam, Kashmiri, Manipuri, Konkani, Maithali, Santhali, Bodo, Dogari, and English. The 12 major scripts used to write these languages are Devanagari, Bangla, Oriya, Gujarati, Gurumukhi, Tamil, Telugu, Kannada, Malayalam, Manipuri, Roman, and Urdu. Of these, Urdu is derived from the Persian script and is written from right to left. The first 10 scripts are originated from the early Brahmi script (300 BC) and are also referred to as Indic scripts. Due to the fact that any optical character recognition (OCR) system can recognize only a script of particular type, it is impossible to design a single recognizer that can identify a variety of scripts/languages. An alternate solution to handle this problem is to use a pool of OCRs corresponding to different scripts. This, in turn, requires the identification of the document’s script before feeding it to the corresponding OCR system. Here lies the problem of script identification. In general, script identification can be achieved at any of the three levels: (a) page level, (b) text-line level, and (c) word level. Identifying scripts at page level can sometimes be too convoluted and protracted due to large computational complexity. Again, the identification of scripts at text-line or word level requires the accurate segmentation of the document pages into their corresponding text-lines and words. That is, the accuracy of the script identification, in turn, depends on the accuracies of its text-line and word segmentation methods, respectively. In addition, identifying text words of different scripts with only a few numbers of characters may not always be feasible because at word level, the number of characters present in a single word may not be always informative. Thus, in the present work, we have performed script identification at block level, where entire document pages are decomposed gradually into their corresponding smaller subblocks using a quad-tree-based page subdivision approach.

A comprehensive literature survey on Indic script identification (described in Ref. [34]) indicates that a lot of researchers have contributed to the handwritten script identification problem by inventing/deriving a bundle of features; however, the exploration of the optimum feature set for this problem has not received any attention from the research community till date in spite of its importance. In general, it is noticed that heuristic search techniques, namely GA, PSO, ACO, and TS, have mostly been applied to reduce the complexity of the FS process. However, in this paper, an HS-based FS procedure is described instead, owing to its simplicity and stochastic behavior (which helps in escaping from trapping at local optima). In comparison with GA-, ACO-, TS-, and PSO-based searches, HS is less explored by the research community, although it has gained huge popularity in recent times. The present HS-based search is also compared with some preceding entrenched optimization methods, and the experimental results also confirm the effectiveness of HS-based search.

4 Proposed Work

The proposed work employs a two-pass approach. In the first pass, the texture-based features are extracted by applying Modified log-Gabor filters [33] designed at multiple scales and multiple orientations. These features are estimated at different sub-image levels of the document images decomposed by the quad-tree-based approach. This approach is recursively applied to subdivided image segments until the image segment does not provide any significant information. Thus, at the first level, the whole document image is divided into four equal blocks (L₁, L₂, L₃, and L₄), whereas at the second level, L₁ block is again subdivided into another four subblocks, namelyL₁₁, L₁₂, L₁₃, and L₁₄. Similarly, each of the other blocks, i.e. L₂, L₃, and L₄, are also subdivided into four subblocks. Thus, at second-level decomposition, a total of 16 blocks is realized. Therefore, the total number of subblocks at l-th level can be written as 4^l, where l denotes the level of decomposition. For an image of size M×N, the size of each subblock at l-th level is (M/2^l)×(N/2^l). For the present work, the maximal value of l is chosen to be 4. Figure 1 shows the quad-tree decomposition for a handwritten Tamil document image. It is evident from Figure 1E that the text block at fourth level has just half the word images of two consecutive text-lines. If the page decomposition is performed beyond fourth level, we would be left with hardly one or two characters of the word image, which may not be sufficient for the script identification purpose. Thus, we limit our quad-tree-based approach for page decomposition till l=4.

Figure 1:

Illustration of (A) Handwritten Tamil Document Page after (B) First-Level (L₁), (C) Second-Level (L₁₁), (D) Third-Level (L₁₁₁), and (E) Fourth-Level (L₁₁₁₁) Decomposition.

In the second pass, the HSA is applied to select the feature subset where the objective function is the performance of the SVM classifier. The block diagram of the proposed two-pass approach is shown in Figure 2.

Figure 2:

Schematic Diagram of the Proposed Two-Pass Approach for Handwritten Script Identification.

5 HS Optimization

HS is a music-based meta-heuristic optimization algorithm, proposed by Geem [12] in 2000. It was inspired by the observation that the aspiration of a musician is to search for a perfect state of harmony. The search process in optimization can be compared to a jazz musician’s improvisation process. Likewise, an optimal solution to an optimization problem should be the desirable solution available to the problem under the given objectives and limited by constraints. Both processes intend to produce the best or optimum output. In order to explain the HS in more detail, let us first understand the improvisation process by a skilled musician. When a musician is improvising, he or she has three possible choices: (a) play any famous piece of music (a series of pitches in harmony) exactly from his or her memory, (b) play something similar to a known piece (thus adjusting the pitch slightly), or (c) compose new or random notes. Geem et al. [13] formalized these three options into a quantitative optimization process in 2001, and the three corresponding components became usage of harmony memory (HM), pitch adjusting, and randomization.

The usage of HM ensures that the best harmonies pass over to the new HM. This usage is controlled by assigning a parameter δ, known as HM accepting or considering rate. The value of δ normally lies between 0 and 1, but typically, it is set to 0.7. The second component called pitch adjustment, ρ∈[0.1, 0.5], is used to control the degree of the adjustment, and for most applications, this is set to 0.3. The third component called randomization is used to increase the diversity of the solutions. Although adjusting pitch has a similar role with randomization, it is limited to certain local pitch adjustment and thus corresponds to a local search. The use of randomization can drive the system further to explore diverse solutions so as to find the global optimality. It is given by

Pitch adjustment is determined by a pitch bandwidth P_band and a pitch adjusting rate ρ. The linear adjustment of pitch is done in the following way:

where x_old is the existing pitch or solution from the HM and x_new is the new pitch after the pitch adjusting action. This essentially produces a new solution around the existing quality solution by varying the pitch slightly by a small random amount. Here, ε is a random number generator in the range of [−1, 1].

5.2 HSA-Based Feature Subset Selection

The HSA consists of the following five steps [24]:

Step 1: Initialization of the Optimization Problem and Algorithm Parameters

The optimization problem can be defined as follows:

Maximize f(x) subject to x_i∈X, X=x₁, x₂, … … ….. x_n, where f(x) is the objective function, X_i is the set of possible range of values for each decision variable, that is Lxi<Xi<Uxi, where Uxi and Lxi are the upper and lower bounds of each decision variable. The control parameters of HS are the HM size (HMS), i.e. the number of solution vectors in the HM (in each iteration), and the maximum number of iterations (Maxltr) or stopping criterion.

Step 2: HM Initialization

HM is a memory location where all the solution vectors (sets of decision variables) are stored. The HM matrix is initially filled with as many randomly generated solution vectors as the HMS.

(13)HM=[x11x21xN1x12x22xN2x1HMSx2HMSxNHMS]

Step 3: New Harmony Improvisation

In this step, a new harmony vector x′=(x′1, x′2, x′3, x′3, … … …, x′N) is generated based on three rules: (a) memory consideration, (b) pitch adjustment, and (c) random selection. Generating a new harmony is called “improvisation”. In the memory consideration, the value of the first decision variable x′1 for the new vector is chosen from the values already existing in the current HM, i.e. from the set {x11, … …, x1HMS}, with a probability δ. Values of the other decision variables x′2, x′3, x′3, … … …, x′N are also chosen in the same manner. The value of δ, varying in the range 0 and 1, is the rate of choosing one value from the previous values stored in the HM, while (1−δ) is the rate of randomly selecting a new value from the possible range of values.

(14)x′i={xiϵ{xi1, xi2, xi3, … … …., xiHMS}with probability δxiϵXiwith probability(1−δ)

Every component obtained by the memory consideration is further examined to determine whether it should be pitch adjusted or not. This operation uses the parameter ρ as follows:

(15)x′i={(x′i±Pband∗ε)with probability ρx′iwith probability(1−ρ)

where P_band is the pitch bandwidth and ε is a random number generator in the range of [−1, 1].

Step 4: Update HM

If the new harmony vector, x′=(x′1, x′2, x′3, x′3, ………, x′N), is better than the worst harmony in the HM, from the point of view of objective function value, the new harmony is included in the HM and the existing worst harmony is excluded from the same.

Step 5: Repetition of Steps 3 and 4 Until the Stopping Criterion Is Satisfied

If the stopping criterion (i.e. maxItr) is satisfied, the computation is terminated. Otherwise, Steps 3 and 4 are repeated.

This is summarized in the flowchart shown in Figure 3.

Figure 3:

Flowchart of the HSA Procedure.

6 Results and Analysis

The experiments are carried out on 12 official scripts of India, namelyBangla, Devanagari, Gujarati, Gurumukhi, Kannada, Malayalam, Manipuri, Oriya, Tamil, Telugu, Urdu, and Roman. All the experiments are implemented in MATLAB 2010 (The Mathworks, Natick, MA, USA) under a Windows XP (Microsoft, Redmond, WA, USA) environment on an Intel Core2 Duo 2.4 GHz processor with 2 GB of RAM. A total dataset of 120 handwritten document pages (10 pages from each script) is collected from different people of varying age, sex, educational qualification, etc. These documents are digitized using an HP flatbed scanner and stored as 24-bit bitmap images. The gray-tone images may contain noisy pixels that have been removed by using Gaussian filter [14]. A set of 160, 640, and 2560 text blocks for each document of different scripts are prepared by applying second-level, third-level, and fourth-level quad-tree-based page segmentation approaches. Sample handwritten text block images of second-level decomposition for all the above-mentioned scripts are shown in Figure 4. A 3-fold cross-validation scheme is used for testing the present approach. That is, for the second-level decomposition approach, a total of 1280 text blocks are used for the training purpose and the remaining 640 text blocks are used for the testing purpose. Similarly, for third-level decomposition, 5120 text blocks are taken for the training and the remaining 2560 text blocks are used for testing the system. Again, for fourth-level decomposition, a set of 20,480 and 10,240 text blocks are utilized for training and testing the present system, respectively. The proposed approach is then evaluated using six different conventional classifiers, namely Naïve Bayes, Multi Layer Perceptron (MLP), SVM, Random Forest, Bagging, MultiClass classifier. The graphical comparison of the identification accuracies of the said classifiers achieved with Modified log-Gabor filter-based features for second-level, third-level, and fourth-level page-segmentation approaches are shown with the help of a bar chart in Figure 5. It can be seen from the figure that the best identification accuracy is achieved by the SVM classifier, which is found to be 92.19%, 90.08%, and 89.26% for second-level, third-level, and fourth-level decomposition, respectively.

Figure 4:

Samples of Text Block Images, Obtained at Second-Level Quad-Tree-Based Decomposition Scheme, Written in (A) Bangla, (B) Devanagari, (C) Gujarati, (D) Gurumukhi, (E) Kannada, (F) Malayalam, (G) Manipuri, (H) Oriya, (I) Tamil, (J) Telugu, (K) Urdu, and (L) Roman scripts.

Figure 5:

Graph Showing the Recognition Accuracies of Modified Log-Gabor Filter-Based Feature Extraction Method for Second-Level, Third-Level, and Fourth-Level Decomposition Approaches Using Six Different Classifiers.

However, it is difficult to decide the optimum set of features that contributed positively in the above script identification experiments. Thus, the HS-based FS method is tested by selecting 20%, 40%, 60%, and 80% of the original feature vector (rounding off to the lowest nearest integer). Here, the objective function is the obviously the classification accuracy, which is evaluated as follows:

As the best classification accuracy has been achieved using the SVM classifier, we have chosen the same for further experimentations. The SVM classifier effectively maps pattern vectors to a high-dimensional feature space where a “best” separating hyperplane (the “maximal margin” hyperplane) is constructed. To construct an optimal hyperplane, SVM employs an iterative training algorithm to minimize an error function, which can be defined as follows:

Find W∈ℝ^m and b∈ℝ to

Minimize  12WTW+C∑i = 1Nξi

subject to constraints y_i(W^T ϕ(x_i)+b)≥1−ξ_i, i=1, 2, …….,N

ξi≥0, i=1, 2… … ., N,

where C is the penalty parameter of the error term, W is the vector of coefficients, b is a constant, and ξ_i represents parameters for handling non-separable data (inputs). The index i labels the N training cases. Note thaty∈±1 represents the class labels and x_i represents the independent variables. The kernel ϕ is used to transform data from the input (independent) to the feature space. It should be noted that the larger the C, the more the error is penalized. Thus, C should be chosen with care to avoid overfitting. There are number of kernels that can be used in SVM models. These include linear, polynomial, radial basis function, and sigmoid. For the present work, we have applied polynomial kernel, which can be written as

where K(X_i, X_j)=ϕ(X_i).ϕ(X_j) is the kernel function that represents a dot product of input data points mapped into the higher-dimensional feature space by transformation ϕ.

For the present work, the optimal values of δ and ρ have been experimentally set to 0.7 and 0.3, respectively. Keeping these values fixed, a comparison between the different values of HMS (HMS=5, 10, 15, and 20) varying the number of iterations (MaxItr=30, 50, and 70) and features (24, 48, 72, and 96) has been performed, which is detailed in Table 1. It can be observed from the table that for HMS=10 and MaxItr=50, the algorithm attains the highest recognition accuracies of 98.18%, 96.56%, and 94.38% for second-level, third-level, and fourth-level decomposition, respectively, which is significantly higher than the preceding outcomes. Apart from this, the optimal number of features required by the HSA is reduced to 72, which comprise almost 60% of the original feature set described in Section 3. In the present work, statistical performance measures with respect to different parameters, namely kappa statistics, mean absolute error, root mean square error, true positive rate, false positive rate, precision, recall, F-measure, and area under ROC are computed for 12 different scripts using the SVM classifier. The said measurements at the second, third, and fourth levels of page decomposition are detailed in Table 2A–C, respectively.

6.1 Sensitivity to Noise

In this subsection, the accuracy of feature subset selection using HSA has been measured by studying the impact of Gaussian noise to the handwritten script document images at all the three levels of decomposition. This is done to validate the robustness of the present methodology in the presence of noise. Presently, we have added 10%, 20%, and 30% Gaussian noises into the pages, and experimented at all the three levels of page decomposition. An example showing the addition of Gaussian noise on a sample handwritten Kannada image at second-level decomposition is illustrated in Figure 6. The Modified log-Gabor filter (as described in Section 3) is again applied to extract the original features, and the number of features is the optimized HSA. The values of parameters of HSA for which the highest identification accuracies have been obtained (in the previous subsection) are taken into consideration for the current experiment. The classification is done using the SVM classifier. A comparative study of the script identification accuracies achieved in the presence of noise at all the three levels of page decomposition are shown with the help of a bar chart in Figure 7. It is to be noted from the figure that the present methodology shows a considerable amount of resistivity to noise. The script recognition accuracies have been found to decrease only about 1%, 2%, and 3% than the present recognition accuracies for 10%, 20%, and 30% addition of Gaussian noise at second-, third-, and fourth-level decompositions, respectively. Thus, it can be concluded from this experiment that the present methodology is less sensitive to noise to a large extent.

Figure 6:

Illustration of (A) Original Handwritten Kannada Image at Second-Level Decomposition, and Addition of Gaussian Noise at (B) 10%, (C) 20%, and (D) 30%.

Figure 7:

Comparison of the Impact of Added Gaussian Noise to Handwritten Script Identification Accuracies (Evaluated Using the SVM Classifier) at All Three Levels of Page Decomposition.

6.2 Statistical Significance Tests

The statistical significance of the experimental setup has also been measured as an essential part for validating the performance of the multiple classifiers using multiple datasets. For statistical comparison of multiple classifiers, two or more learning algorithms are first trained and tested on suitable datasets and then evaluated using the classification accuracies. Datasets are randomly divided into small datasets with different sample sizes. Performances of different classifiers are then carried out for each randomly created dataset. The only requirement for performing non-parametric tests is that the compiled results provide reliable estimates of the performances of the classification algorithms on each dataset. In the usual experimental setups, these numbers come from cross-validation or from repeated stratified random splits onto training and testing datasets. The term “sample size” refers to the number of datasets used, not the number of training/testing samples drawn from each individual set. The sample size can therefore be as small as five and is usually well below 30. In the present work, we have performed some statistical significance tests for validating the results of the best classifier among six different classifiers at second-level decomposition. These tests have been performed at two stages: (a) before applying HSA and (b) after applying HSA. The theoretical discussions related to these tests are already described in one of our previous works [16].

Case I: Application of Statistical Significance Tests prior to HSA

A safe and robust non-parametric Friedman test [14] has been conducted along with the corresponding post-hoc tests. For the experimentation, the number of datasets (N) and the number of classifiers (k) are set as 12 and 6, respectively. These datasets are chosen randomly from the test set. The performances of the classifiers on different datasets are shown in Table 3. On the basis of these performances, the classifiers are then ranked for each dataset separately, the best-performing algorithm gets the rank 1, the second best gets the rank 2, and so on (see Table 3). Average ranks are assigned in case of ties.

Table 3:

Recognition Accuracies of Six Classifiers and Their Corresponding Ranks Using 12 Different Datasets Before Performing HSA (Ranks in Parentheses Are Used for Performing Friedman Test).

Dataset	Classifier
	Naïve bayes (%)	MLP (%)	SVM (%)	Random forest (%)	Bagging (%)	Multiclass classifier (%)
#1	90 (5)	93 (3)	98 (1)	88 (6)	91 (4)	95 (2)
#2	88 (6)	93 (2)	96 (1)	91 (3)	90 (4)	89 (5)
#3	93 (3)	91 (4)	95 (2)	90 (5.5)	90 (5.5)	97 (1)
#4	91 (4.5)	95 (2)	97 (1)	93 (3)	90 (6)	91 (4.5)
#5	90 (6)	95 (2)	96 (1)	93 (4)	91 (5)	95 (3)
#6	96 (2)	90 (5.5)	98 (1)	90 (5.5)	91 (4)	93 (3)
#7	93 (2)	91 (5)	95 (1)	91 (5)	92 (3)	91 (5)
#8	88 (6)	95 (3)	96 (1.5)	92 (4)	90 (5)	96 (1.5)
#9	95 (2)	94 (3)	97 (1)	90 (6)	91 (4.5)	91 (4.5)
#10	92 (2)	86 (5.5)	94 (1)	90 (3)	86 (5.5)	88 (4)
#11	90 (4.5)	90 (4.5)	95 (1)	93 (2)	91 (3)	88 (6)
#12	89 (6)	94 (2.5)	96 (1)	94 (2.5)	90 (5)	92 (4)
Mean rank	R1=4.08	R2=3.5	R3=1.125	R4=4.125	R5=4.54	R6=3.625

1. The mean ranks scored by individual classifiers are shaded in gray and styled in bold.

Let rji be the rank of the j-th classifier on i-th dataset. Then, the mean of the ranks of the j-th classifier over all the N datasets will be computed as

(18)Rj=1N∑i = 1Nrji

The null hypothesis states that all the classifiers are equivalent and so their ranks R_j should be equal. To justify it, the Friedman statistic [16] is computed as follows:

(19)χF2= 12Nk(k+1)[∑jRj2− k(k+1)24]

Under the current experimentation, this statistic is distributed according to χF2 with k−1 (=5) degrees of freedom. Using Eq. (19), the value of χF2 is calculated as 25.47. From the table of critical values (see any standard statistical book), the value of χF2 with 5 degrees of freedom is 11.07 for α=0.05 (where α is known as the level of significance). It can be seen that the computed χF2 differs significantly from the standard χF2. Thus, the null hypothesis is rejected.

Iman et al. [16] have derived a superior statistic using the following formula:

(20)FF=(N−1)χF2N(k−1)−χF2

F_F is distributed according to the F-distribution with k−1 (=5) and (k−1)(N−1) (=55) degrees of freedom. Using Eq. (20), the value of F_F is calculated as 8.115. The critical value of F (5, 55) for α=0.05 is 4.43 (see any standard statistical book), which shows a significant difference between the standard and calculated values of F_F. Thus, both the Friedman and Iman et al. statistics reject the null hypothesis.

As the null hypothesis is rejected, a post-hoc test known as the Nemenyi test [16] is carried out for pair-wise comparisons of the best- and worst-performing classifiers. The performances of two classifiers are significantly different if the corresponding average ranks differ by at least the critical difference (CD), which is expressed as

(21)CD=qαk(k+1)6N

For Nemenyi’s test, the value of q_0.05 for six classifiers is 2.850 (see Table 5a of Ref. [6]). Thus, the CD is calculated as 2.8506∗76∗12, i.e. 2.17 using Eq. (21). As the difference between the mean ranks of the best and worst classifier is much greater than the CD, we can conclude that there is a significant difference between the performing ability of the classifiers. For comparing all the classifiers with a control classifier (here SVM), we have applied the Bonferroni-Dunn test [16]. For this test, CD is calculated using the same Eq. (21). However, here, the value of q_0.05 for six classifiers is 2.576 (see Table 5b of Ref. [6]). Thus, the CD for the Bonferroni-Dunn test is calculated as 2.5766∗76∗12, i.e. 1.96. As the difference between the mean ranks of any classifier and SVM is always greater than CD, the chosen control classifier performs significantly better than other classifiers. A graphical representation of the said post-hoc tests for comparison of six different classifiers is shown in Figure 8.

Figure 8:

Graphical Representation of Comparison of Multiple Classifiers for (A) Nemenyi’s Test and (B) Bonferroni-Dunn’s Test Before Applying HSA.

After performing the above-mentioned statistical significance tests over the 12 datasets and six classifiers, it can be concluded that the SVM classifier outperforms all other classifiers at second-level decomposition. Similarly, the performances of these tests at the remaining two levels also confirm that the SVM classifier is found to be statistically significant than the other five classifiers.

Case II: Application of Statistical Significance Tests subsequent to HSA

Similar to the preceding case, the number of datasets (N) and the number of classifiers (k) are chosen as 12 and 6, respectively. The performances of the classifiers on different datasets are shown in Table 4.

Table 4:

Recognition Accuracies of Six Classifiers and Their Corresponding Ranks Using 12 Different Datasets after Applying HSA (Ranks in Parentheses Are Used for Performing Friedman Test).

Dataset	Classifier
	Naïve bayes (%)	MLP (%)	SVM (%)	Random forest (%)	Bagging (%)	Multiclass classifier (%)
#1	93 (6)	99 (2.5)	100 (1)	97 (5)	99 (2.5)	98 (4)
#2	99 (2.5)	99 (2.5)	100 (1)	96 (5.5)	96 (5.5)	97 (4)
#3	98 (4.5)	93 (6)	100 (1)	99 (2.5)	98 (4.5)	99 (2.5)
#4	93 (6)	98 (3)	99 (1.5)	99 (1.5)	97 (4)	96 (5)
#5	95 (6)	99 (2.5)	100 (1)	99 (2.5)	97 (5)	98 (4)
#6	99 (2.5)	96 (6)	100 (1)	97 (5)	98 (4)	99 (2.5)
#7	91 (6)	99 (3)	100 (1)	99 (3)	99 (3)	98 (5)
#8	93 (6)	98 (3.5)	100 (1)	98 (3.5)	96 (5)	99 (2)
#9	95 (6)	100 (1.5)	100 (1.5)	97 (5)	99 (3.5)	99 (3.5)
#10	99 (2.5)	99 (2.5)	100 (1)	97 (5.5)	97 (5.5)	98 (4)
#11	94 (6)	98 (4.5)	100 (1)	99 (2.5)	98 (4.5)	99 (2.5)
#12	98 (3.5)	97 (5.5)	100 (1)	98 (3.5)	99 (2)	97 (5.5)
Mean rank	R1=4.79	R2=3.58	R3=1.083	R4=3.75	R5=4.08	R6=3.708

1. The mean ranks scored by individual classifiers are shaded in gray and styled in bold.

Using the Friedman statistic given in Eq. (19), the value of χF2 is calculated as 27.057. It can be seen that the computed χF2 differs significantly from the standard χF2. Using the Iman et al. statistic written in Eq. (20), the tabulated value of F_F is calculated as 9.035, which is much greater than the observed value of F_F for (5,55) degrees of freedom. Thus, for both tests, the null hypothesis is rejected.

For Nemenyi’s test, the value of CD is calculated as 2.8506∗76∗12, i.e. 2.17, using the same Eq. (21). As the difference between the mean ranks of the best and worst classifier is much greater than the CD, we can conclude that there is a significant difference between the performing ability of the classifiers. For comparing all the classifiers with a control classifier using the Bonferroni-Dunn test, the value of CD is calculated as 2.5766∗76∗12, i.e. 1.96. As the difference between the mean ranks of any classifier and SVM is always greater than CD, the chosen control classifier performs significantly better than other classifiers. A graphical representation of the above-mentioned post-hoc tests for comparison of six different classifiers is shown in Figure 9.

Figure 9:

Graphical Representation of Comparison of Multiple Classifiers for (A) Nemenyi’s Test and (B) Bonferroni-Dunn’s Test After Applying HSA.

After performing the above-mentioned statistical significance tests over the 12 datasets and six classifiers, it can be concluded that the SVM classifier outperforms all other classifiers at second-level decomposition. Similarly, the performance of these tests at the remaining two levels also confirms that the SVM classifier is found to be statistically significant than the other five classifiers.

7 Conclusion

This paper has investigated the HSA for the selection of the optimal feature subset in handwritten script identification problem. The proposed HSA evolved upon the HM, which adaptively stores updated solutions during the evolution. This algorithm has been evaluated on a limited dataset of script blocks written in 12 official Indian scripts. For obtaining the optimal classification accuracy, we have conducted a series of experiments to test the efficiency of HSA with different parameter settings. The proposed HSA achieves the highest classification accuracy of 98.18% at the second-level block decomposition approach, which is almost about 6% more than the earlier attained result while only considering almost 60% of the original feature set. The improved result justifies the need for selecting the subset of designed features for this type of pattern classification problem, where both local and global features are computed without knowing the exact importance of the features from all regions of a particular script block image. Although the results show a rather encouraging trend, much work can still be done to further increase the potential of the present work.

In particular, investigations regarding how the algorithm parameters may be better tuned are necessary. It would be useful to develop a better stopping criterion, which will, in turn, lessen the computation time. In addition, a method may be devised to dynamically specify the size of the HM, which affects the convergence rate and search parameters.