A novel associative classification model based on a fuzzy frequent pattern mining algorithm

https://doi.org/10.1016/j.eswa.2014.09.021Get rights and content

Highlights

  • We propose a novel efficient fuzzy associative classification approach.

  • We exploit a fuzzy version of the FP-Growth algorithm.

  • We perform an experimental analysis on 17 classification datasets.

  • We compare our approach with three well-known associative classifiers.

Abstract

Associative classification models are based on two different data mining paradigms, namely pattern classification and association rule mining. These models are very popular for building highly accurate classifiers and have been employed in a number of real world applications.

During the last years, several studies and different algorithms have been proposed to integrate associative classification models with the fuzzy set theory, leading to the so-called fuzzy associative classifiers.

In this paper, we propose a novel efficient fuzzy associative classification approach based on a fuzzy frequent pattern mining algorithm. Fuzzy items are generated by discretizing the input variables and defining strong fuzzy partitions on the intervals resulting from these discretizations. Then, fuzzy associative classification rules are mined by employing a fuzzy extension of the FP-Growth algorithm, one of the most efficient frequent pattern mining algorithms. Finally, a set of highly accurate classification rules is generated after a pruning stage.

We tested our approach on seventeen real-world datasets and compared the achieved results with the ones obtained by using both a non-fuzzy associative classifier, namely CMAR, and two recent state-of-the-art classifiers, namely FARC-HD and D-MOFARC, based on fuzzy association rules. Using non-parametric statistical tests, we show that our approach outperforms CMAR and achieves accuracies similar to FARC-HD and D-MOFARC.

Introduction

Pattern classification and association rule mining are two of the most studied data mining paradigms (Witten & Frank, 2011). Pattern classification deals with assigning a class label to an object described by a set of features. The classification task is carried out by using a specific model, namely the classifier, previously built by using a set of training examples. Association rule mining is the task of discovering correlation or other relationships among items in large database (Agrawal, Imieliński, & Swami, 1993).

During the last years, association rule mining has become a very popular method to build highly accurate classification models. Such method is able to mine a set of high quality classification rules from huge amounts of data and to achieve a considerable performance in terms of classification accuracy. Associative classification, that is, classification based on association rules, has been extensively studied in the literature (Li et al., 2001, Yin and Han, 2003, Chen and Hung, 2009, Baralis and Garza, 2012, Abdelhamid et al., 2012) and has been recently exploited in a number of real world applications such as detection phishing activities in websites (Ajlouni, Hadi, & Alwedyan, 2013), classification of XML documents (Costa, Ortale, & Ritacco, 2013), text analysis (Yoon & Lee, 2013) and classification of medical diseases (Dua et al., 2009, Jabbar et al., 2013).

The hybridization of the two data mining paradigms can be summarized in the following steps, which characterize the generation and the use of an associative classifier. First, a set of classification association rules (CARs) is mined from the training set. Second, a rule pruning step discards redundant or noisy information contained in the rule set and selects a subset of high quality CARs. The selected CARs are used to predict the class labels when the model is used for classifying unlabeled patterns.

As stated in Pach, Gyenesei, and Abonyi (2008), even though learning based on association rule mining ensures high accuracy in pattern classification and generates rule-based models that are often “interpretable” by the user, this model suffers from some main weaknesses. First, when the number of training data objects is huge, the complexity of the learning process grows exponentially in terms of both time and memory. Second, association rule mining algorithms deal with binary or categorical itemsets. On the other hand, real data objects are often described by numerical continuous features. Thus, appropriate discretization algorithms have to be applied to transform continuous feature domains into a set of items.

As regards the issue of managing continuous input variables, associative classification approaches adopt discretization algorithms for extracting a set of items and therefore for allowing the rule mining algorithms to work properly. The discretization is accomplished by assigning each value to a bin. The data ranges (bin boundaries) and the number of bins are determined by the discretization algorithm. Bin boundaries are typically crisp, but crisp discretization is not natural. Indeed, the transitions between bins are not generally abrupt, but rather gradual. Thus, fuzzy sets are certainly more appropriate for describing attribute partitions. In fact, in the last years, a number of associative classification approaches have used fuzzy boundaries, thus generating fuzzy association rules (Pach et al., 2008, Chen and Chen, 2008, Alcala-Fdez et al., 2011, Lucas et al., 2012, Orriols-Puig et al., 2013, Ma et al., 2014, Fazzolari et al., 2014).

In this paper, we propose a new efficient fuzzy association rule-based classification scheme, which mines fuzzy CARs by using a fuzzy version of the well-known FP-Growth algorithm (Han, Pei, Yin, & Mao, 2004). Even though some fuzzy versions of FP-Growth have been already proposed in the literature (Lin et al., 2010, Wang et al., 2010), to the best of our knowledge, our method represents the first attempt of using such algorithm for deriving fuzzy CARs. Indeed, the works in (Lin et al., 2010, Wang et al., 2010) just propose a fuzzy version of the FP-Growth algorithm aimed at mining fuzzy association rules for descriptive modeling rather than for classification.

The use of fuzzy partitions makes the fuzzy CAR mining more complex. Indeed, while in the case of crisp partitions an input value supports a unique item, in the case of fuzzy partitions, an input value can support more than one fuzzy item (in our implementation, which is based on strong fuzzy partitions, the supported items are at most two). Thus, the number of possible fuzzy association rules is higher than the number of possible crisp rules. The approaches proposed so far in the literature for generating fuzzy association rules have limited the complexity by considering only the most frequent fuzzy item for each attribute (Lin et al., 2010, Wang et al., 2010). Obviously, this solution reduces the number of association rules, but also the amount of information described by these rules. In this context, we aim to exploit the advantages of fuzzy set theory in terms of modeling capability, without dramatically reducing the complexity and therefore the information. Thus, we have proposed a set of appropriate novel strategies, which allow us to efficiently generate accurate fuzzy associative classifiers. In particular, the main novelties introduced in the proposed fuzzy association rule-based classification scheme are:

  • A novel approach to define strong fuzzy partitions from crisp partitions obtained by applying the classical Fayyad and Irani discretization algorithm (Fayyad & Irani, 1993).

  • The extension of the FP-Growth algorithm to the fuzzy context for mining a set of fuzzy CARs. In particular, we adopt proper definitions of fuzzy support and confidence. Further, we consider, for each attribute, all the frequent fuzzy sets rather than only the most frequent when generating the fuzzy CARs. Finally, we just adopt the FP-Growth algorithm. Most of the proposed fuzzy associative classifiers, such as the ones described in (Pach et al., 2008, Chen and Chen, 2008, Alcala-Fdez et al., 2011, Lucas et al., 2012, Fazzolari et al., 2014), are based on the Apriori algorithm. This algorithm, as discussed in Section 2, is characterized by a number of weaknesses, especially when dealing with large and high-dimensional datasets.

  • Three purposely adapted types of fuzzy CAR pruning. The first type considers fuzzy support and confidence with respect to two thresholds, namely minSupp and minConf. These thresholds are adapted to the number of conditions and number of instances of each class, respectively, so as to take into account the effect of the specific implementation of the conjunction operator and the imbalance of datasets. The second type removes redundant rules based on fuzzy support and confidence, and rule length. The third type exploits the training set coverage: only the fuzzy rules, which are activated by at least one data object in the training set, are retained.

  • An adjustment of the weighted vote reasoning method for classifying unlabeled patterns: the vote of each single rule is modified accordingly to its rule length. This modification balances the relevance of more general and more specific rules, thus improving the overall classification accuracy.

We compare the results achieved by the proposed approach on seventeen datasets with the ones obtained by CMAR (Li et al., 2001), an associative classifier based on the FP-Growth algorithm. By using non-parametric statistical tests, we show that our approach outperforms CMAR in terms of accuracy. Further, we compare the proposed fuzzy associative classifier with two recent state-of-the-art approaches, namely FARC-HD (Alcala-Fdez et al., 2011) and D-MOFARC (Fazzolari et al., 2014), for mining fuzzy CARs. We show that our approach is statistically equivalent to the comparison approaches. On the other hand, we have to highlight that FARC-HD and D-MOFARC employ a fuzzy adaptation of the Apriori algorithm for mining the fuzzy rules and an evolutionary post-processing for pruning these rules and optimizing the fuzzy partitions. Thus, our fuzzy associative classification scheme, based on the fuzzy FP-Growth, results to be more scalable, especially when dealing with large and high dimensional datasets.

This paper is organized as follows. Section 2 discusses some related works in the framework of associative classification. Section 3 provides a basic description of the fuzzy rule-based classifiers and their inference models, and introduces some notations for the fuzzy CARs. Section 4 describes each phase of the proposed approach and includes the details of the fuzzy FP-Growth algorithm. Section 5 presents the experimental setup and discusses the results that are obtained on seventeen real-world datasets. Finally, in Section 6, we draw some final conclusions.

Section snippets

Related works

Association rule mining is crucial to the success of the associative classification models. This mining process is generally performed in three steps. First, frequent itemsets are extracted from the training set. An itemset is frequent when its occurrence in the training set is higher than a prefixed threshold. Then, rules are mined from the frequent itemsets. Finally, rules are pruned by, for instance, considering their confidence and/or redundancy. The identification of fast and efficient

Fuzzy rule-based classifiers

In this section, we first describe the structure of fuzzy rule-based classifiers (FRBCs) and the inference model adopted for classifying patterns. Then, we introduce some notations for fuzzy association rules for classification.

The proposed approach

In this section, we present our associative classifier based on a fuzzy frequent pattern (AC-FFP) mining algorithm. AC-FFP consists of the following three phases:

  • 1.

    Discretization: a fuzzy partition is defined on each linguistic variable by using the multi-interval discretization approach based on entropy proposed by Fayyad and Irani (1993).

  • 2.

    Fuzzy CAR Mining: a fuzzy frequent pattern mining algorithm, which is an extension of the well known FP-Growth, is exploited to extract frequent fuzzy

Experimental study

We tested our method on seventeen classification datasets extracted from the KEEL repository (available at http://sci2s.ugr.es/keel/datasets.php). As shown in Table 4, the datasets are characterized by different numbers of input variables (from 4 to 16), input/output instances (from 106 to 19020) and classes (from 2 to 11). For the datasets CLE and WIS, we removed the instances with missing values. The number of instances in the table refers to the datasets after the removing process.

We compare

Conclusions

In this paper, we have proposed a new efficient fuzzy association rule-based classification scheme based on a fuzzy version of the well-known FP-Growth algorithm.

The development of the classification scheme has required to introduce a number of purposely-defined strategies for: (i) appropriately generating the fuzzy partitions, (ii) extending the FP-Growth algorithm to the fuzzy context, (iii) selecting the most accurate and non-redundant fuzzy rules and (iv) performing the classification of

References (48)

  • C.-W. Lin et al.

    Linguistic data mining with fuzzy FP-trees

    Expert Systems with Applications

    (2010)
  • A. Orriols-Puig et al.

    Unsupervised KDD to creatively support managers’ decision making with fuzzy association rules: A distribution channel application

    Industrial Marketing Management

    (2013)
  • F.P. Pach et al.

    Compact fuzzy association rule-based classifier

    Expert Systems with Applications

    (2008)
  • A.Y. Rodriguez-Gonzalez et al.

    Mining frequent patterns and association rules using similarities

    Expert Systems with Applications

    (2013)
  • B. Vo et al.

    A lattice-based approach for mining most generalization association rules

    Knowledge-Based Systems

    (2013)
  • Y. Yoon et al.

    Two scalable algorithms for associative text classification

    Information Processing and Management

    (2013)
  • N. Abdelhamid et al.

    MAC: A multiclass associative classification algorithm

    Journal of Information & Knowledge Management

    (2012)
  • R. Agrawal et al.

    Mining association rules between sets of items in large databases

    SIGMOD Record

    (1993)
  • R. Agrawal et al.

    Fast algorithms for mining association rules in large databases

  • M.I.A. Ajlouni et al.

    Detecting phishing websites using associative classification

    European Journal of Business and Management

    (2013)
  • J. Alcala-Fdez et al.

    A fuzzy association rule-based classification model for high-dimensional problems with genetic rule selection and lateral tuning

    IEEE Transactions on Fuzzy Systems

    (2011)
  • J. Alcalá-Fdez et al.

    Keel: A software tool to assess evolutionary algorithms for data mining problems

    Soft Computing

    (2009)
  • Baralis, E., & Garza, P. (2002). A lazy approach to pruning classification rules. In Proceedings of 2002 IEEE...
  • E. Baralis et al.

    I-prune: Item selection for associative classification

    International Journal of Intelligent Systems

    (2012)
  • Cited by (39)

    • ARC-SL: Association rule-based classification with soft labels

      2021, Knowledge-Based Systems
      Citation Excerpt :

      Then, the method is extended to the multi-objective context by designing new fuzzy partition and genetic post-processing stages in [50]. After that, the AC-FFP [51] is proposed by using the fuzzy version of FP-Growth algorithm for efficiently mining frequent rule-items. To handle the uncertain information in complex environment, the BRBCS [52] is presented for classification by introducing a belief rule structure with fuzzy antecedent and belief consequent.

    • EARC: Evidential association rule-based classification

      2021, Information Sciences
      Citation Excerpt :

      Several association rule mining techniques have been adopted to list frequent rule-items from training instances, in which the most popular one is the Apriori algorithm [3], due to its simplicity and good performance especially when the size of the candidate rule-items is not very large. Further, to avoid the unnatural boundaries in the partition of continuous features, the idea of integrating association rule mining with fuzzy rule-based classification was first proposed in [26], and then different studies have been carried out to obtain fuzzy association rule-based classifiers [25,15,6,9]. A key characteristic of the belief rule structure, including both the evidential CAR and belief classification rule, is that the rule consequent is in the form of belief distribution.

    • An improved rule induction based denial of service attacks classification model

      2020, Computers and Security
      Citation Excerpt :

      In the first phase, the algorithm uses the training dataset for generating a set of rules which all have accuracy > Rule-Power. Rule-Power is like the confidence parameter that is utilized in association classification approach (Antonelli et al., 2015) whose goal is to come up with best possible rule set in addition to rules that have 100% accuracy rate as in PRISM. The suggested training strategy assures the creation of rule set which have 0 error-rate in addition to rules which pass the Rule-Power threshold.

    View all citing articles on Scopus
    View full text