High Relevance Keyword Extraction facility for Bayesian text classification on different domains of varying characteristic

Abstract

High Relevance Keyword Extraction (HRKE) facility is introduced to Bayesian text classification to perform feature/keyword extraction during the classifying stage, without needing extensive pre-classification processes. In order to perform the task of keyword extraction, HRKE facility uses the posterior probability value of keywords within a specific category associated with text document. The experimental results show that HRKE facility is able to ensure promising classification performance for Bayesian classifier while dealing with different text classification domains of varying characteristics. This method guarantees an effective and efficient Bayesian text classifier which is able to handle different domains of varying characteristics, with high accuracy while maintaining the simplicity and low cost processes of the conventional Bayesian classification approach.

Introduction

Text document classification denotes the task of assigning raw text documents to one or more pre-defined categories. This is a direct concept from machine learning, which implies the declaration of a set of labelled categories as a way to represent the documents, and a statistical classifier trained with a labelled training set. Classification is the process in which objects are initially recognized, differentiated and understood, and implies that objects are grouped into categories, usually for some specific purposes. Ideally, a category represents a relationship between the subject and the object of knowledge. Classification is fundamental in prediction, inference, and decision making. However, there are a variety of ways to approach classification task. An increasing number of supervised classification approaches have been developed for various types of classification tasks, such as rule induction (Apte et al., 1994, Provost, 1999), k-nearest neighbor classification (Han, Karypis, & Kumar, 1999), maximum entropy (Nigam, Lafferty, & McCallum, 1999), artificial neural network (Diligenti et al., 2003a, Diligenti et al., 2003b), support vector machines (Isa et al., 2008a, Isa et al., 2008b, Joachims, 1998, Lin, 1999), and Bayesian classification (Domingos and Pazzani, 1997, Eyheramendy et al., 2003, Kim et al., 2002, McCallum and Nigam, 2003, O’Brien and Vogel, 2003, Provost, 1999, Rish, 2001). Besides the supervised classification approaches, the unsupervised clustering approaches, such as self-organizing map (Adami et al., 2005, Hartley et al., 2006, Isa et al., 2009, Wang, 2001) have also been widely implemented in segmenting data into groups for further analysis and processing. Among these approaches, Bayesian classification has been widely implemented in many real world applications due to its relatively simple training and classifying algorithms.

One of the outstanding features of Bayesian classification as compared to other classification approaches is its ability and simplicity in handling raw text data directly, without requiring any pre-process to transform text data into a representation suitable format, typically in numerical form, as required by most of the successful and highly accurate text classification approaches, such as by the use of k-nearest neighbour (k-NN) and support vector machines (SVM) classifiers. As a trade-off to its simplicity, Bayesian classification has been reported as one of the poorest-performing classification approaches by many research groups through extensive experiments and evaluations (Brücher et al., 2002, Yang and Liu, 1999). In order to enhance the performance of Bayesian classifier, researchers have proposed several pre-processes such as stop word elimination, word stemming, and feature selection methods (Al-Mubaid and Umair, 2006, Apte et al., 1994, Chen et al., 2009, Dhillon et al., 2002, Eyheramendy et al., 2003, Han et al., 1999; Isa et al., 2008a, Isa et al., 2008b; Joachims, 1998, Joachims, 1999, Kim et al., 2002, McCallum and Nigam, 2003, Ozgur et al., 2005, Yang and Pedersen, 1997, Yang and Liu, 1999). However, by implementing the pre-processes, the simplicity and low cost training and classifying algorithms of Bayesian classification would have to be sacrificed due to the pre-processing stages which consume high time, physical memory and CPU usages, and also require extensive human expert interaction. It is the goal of this paper to enhance the classification effectiveness of Bayesian text classification, while maintaining the simplicity and low cost training and classification processes. In this paper, we introduce the High Relevance Keyword Extraction (HRKE) facility which uses a simple algorithm to perform stop word elimination and feature selection during the classifying stage, without the need for any pre-process and additional human experts’ involvement.

Conventional Bayesian classification takes the entire body of text document into account for training and classifying purposes. Due to the fact that text documents contain irrelevant words, the accuracy of Bayesian classification is severely degraded by the presence of noisy or irrelevant features. Extensive research works have been carried out in order to counter the problem above by introducing pre-processes such as stop word elimination and feature selection to be applied to the training set and the testing set in order to eliminate irrelevant and low informative features, and in the context of text classification, the low relevance keywords.

Stop word elimination is the procedure where common words which can be seen in many documents such as “a”, “an”, “the”, “to”, “for”, “be”, etc. are eliminated from the documents contained in the dataset (Baeza-Yates & Ribeiro-Neto, 1999). To perform stop word elimination, a list of stop words is used to match each individual word from the text documents. The words that contained in text documents which match any word from the list of stop words will not be taken into account for both the training and classifying processes. There is a potential drawback of stop word elimination, where certain words which are considered as stop words for a particular dataset (domain), but can be highly informative features for another dataset (domain) (Takamura, 2003).

Besides the simple stop word elimination technique, there are several statistical methods for feature selection which have been introduced as pre-processes for Bayesian text classification. These methods provide a measure for usefulness of each individual word in the classification task. Some of the common statistical feature selection methods are document frequency thresholding, information gain, mutual information, x² statistic, and term strength. The feature selection methods mentioned above are discussed and compared in Yang and Pedersen (1997). Besides these, there are also some feature selection methods which have been invented specifically for Bayesian classification such as Multi-class Odds Ratio (MOR), and Class Discrimination Measure (CDM) which have been experimentally proven to be better than other feature selection approaches (Chen et al., 2009).

It was found here that most of the feature selection methods are carried out as pre-processes prior to the classification process. Furthermore, the pre-processes consume additional time, memory and CPU usages, hence the efficiency of the classification task is degraded. It is the goal of the work presented in this paper to introduce a keyword extraction facility to perform the feature selection during the classifying stage of the classifier, without needing any extensive and costly pre-process. This keyword extraction facility implements Bayesian probabilistic algorithm to determine the “Importance” of keywords based on their posterior probability values being annotated to each of the available categories, during the classifying stage performed by Bayesian classifier. Only the “Important” keywords are taken into consideration as the features for the classification and this contributes to an effective and efficient classification, by performing the feature selection during the process of classification.