Comparison of different input selection algorithms in neuro-fuzzy modeling

https://doi.org/10.1016/j.eswa.2011.08.049Get rights and content

Abstract

Data driven neuro-fuzzy systems modeling requires the application of a suitable input selection method to identify the most relevant input variables. In view of the substantial number of existing input selection algorithms applied in neuro-fuzzy modeling, the need arises to count on criteria that enable to adequately decide which algorithm to use in certain situations. In this paper, we analyze the performance of five fundamental and widely used input selection algorithms, which encompass both model-free methods and model-based methods. Each of these algorithms is discussed in detail, and thus, present a comprehensive comparative analysis. Finally, we compare the performances of these algorithms by applying in stock price prediction problem. The experiments and the results provide a precious insight about the advantages and drawbacks of these five input selection algorithms.

Highlights

► Need to count on criteria that enable to decide which algorithm to use in certain situations. ► The performance of five fundamental and widely used input selection algorithms have compared. ► The results provide a precious insight about the advantages and drawbacks of each algorithm.

Introduction

As any area matures, there is the need to better understand its elements and different proposed algorithms for each element. Neuro-fuzzy system modeling is one of the most important issues in fuzzy sets theory. In neuro-fuzzy modeling, fuzzy rules approximate the highly non-linear relations between input–output data. Earlier, these fuzzy rules were created by means of domain experts’ knowledge, which was very subjective and volatile. To overcome these problems, many efforts arose on modeling approaches for data-driven system identification in terms of fuzzy if–then rules (Castellano et al., 2005, Sugeno and Yasukawa, 1993, Uncu et al., 2003).

Data driven neuro-fuzzy systems modeling consists of two basic steps: system (structure) identification, and fuzzy reasoning. System identification step includes input selection, fuzzy if–then rules generation, and the model’s parameters selection. Fuzzy reasoning is concerning about extracting new knowledge from the rule-based systems using inference methodologies. Use of the fuzzy reasoning is one the most important advantages of neuro-fuzzy modeling approach, as it involves all rules in creating the final result.

All learning systems such as neural networks and fuzzy systems suffer from complexity and noise data disturbance. Modeling real world problems deals with a large number of candidates input variables, and it increasingly leads to high degree of system’s complexity. On the other hand, because of the high correlation between potential input variables in addition to non-linear relation with the output variable, the results of classical tools such as principal component analysis and least square method are unreliable. Input selection is thus a crucial step with the aim of Sindelár and Babuška (2004): (1) reducing the model’s complexity, (2) removing correlated inputs, and (3) not having contribution to the output, removing noise inputs.

Generally all input selection methods used in neuro-fuzzy modeling consist of four steps (Hu & Wan, 2009): the generation of candidate subsets, subsets evaluation and selection, stopping criteria, and model validation. From the use of different evaluation functions point of view, input selection methods found in the literature can generally be divided into two main categories (Sindelár & Babuška, 2004): model-free methods, and model-based methods. In model-free methods there is no need to develop a model for input selection procedure. These methods only use the data set and work through principal component analysis, factor analysis, properties of functions, branch and bound procedures, etc. He and Asada (1993) exploit the continuity proper of non-linear functions. In order to find the optimal order of input–output model, they computed the “Lipschitz” coefficients. Poncet and Moschytz (1994) proposed a method based on estimating the model performance directly from data. Later, Sindelár and Babuška (2004) presented a method for the selection of significant inputs in non-linear regression models. Given a set of input–output data and an initial superset of potential inputs, their model selects the relevant inputs by checking whether after deleting a particular input, the data set is still consistent with the basic property of a function or not. In another work, using backward evolutionary selection of the terms in a polynomial model, Maertens, Baerdemaeker, and Babuška (2006) introduced an algorithm to rank the candidate input variables as possible regressor variables for the prediction of specific output.

As oppose to model-free input selection methods, model-based methods use a particular model to find the most important inputs. Linear models often use the Akaike’s information criterion (Ljung, 1999). Models with different sets of input variables are compared and the model with minimum value of the Akaike information criterion is selected.

On the other hand, in non-linear systems, input selection is usually done based on heuristic criteria. A more common way is to compare all combinations of inputs by using a predefined evaluation criterion. Jang (1993) applied this approach to select the most relevant inputs for adaptive neuro-fuzzy inference system (ANFIS) learning. He applied a single epoch of training for each combination of input variables. Although this method was developed for the ANFIS fuzzy system, the same idea can possibly be used for other neuro-fuzzy systems. In the same year, the so called “regularity criterion” which is based on dividing a data set into two parts was used by Sugeno and Yasukawa (1993). Lin (1993) proposed a method based on “fuzzy curves” that represented the sensitivity of the output with respect to the inputs. Later, Nakashima, Morisawa, and Ishibuchi (1997) presented a stepwise input selection mechanism; first the most important two attributes are selected by the examination of all combinations, and then the next best attribute is added sequentially.

Contrary to the already proposed clustering-based methods, Linkens and Chen (1999) proposed an input selection algorithm in which inputs are determined on the basis of a class of sub-clusters created by a self-organizing network instead of the data. The important input variables which independently and significantly influence the system output can be extracted by a fuzzy neural network. Gaweda, Zurada, and Setiono (2001) presented an iterative backward selection method for determination of relevant input variables in data-driven fuzzy modeling. Their method utilizes parameters of the Takagi–Sugeno model as a factor to determine the significance of input variables. Uncu and Turksen (2007) introduced a feature selection algorithm that combines feature wrapper and feature filter approaches to identify the significant input variables in the systems with continuous domains. The proposed method utilizes functional dependency concept, correlation coefficients and K-nearest neighborhood method to implement the feature filter and feature wrappers. Vieira, Sousa, and Kaymak (2009) proposed a fuzzy objective function to cope with the difficulty of weighting the different criteria involved in the multi-objective optimization algorithm of feature selection. More recently, Vieira, Sousa, and Runkler (2010) proposed an algorithm for feature selection based on two cooperative ant colonies, which minimizes two objectives: the number of features and the classification error. They used two pheromone matrices and two different heuristics for these objectives.

All input selection approaches and algorithms have their own advantages and disadvantages. The objective of this paper is to explain and compare the features and performance of five fundamental and widely used input selection methods in neuro-fuzzy systems modeling. At the moment, a comparison is not available in the literature, specially for the use in financial time series prediction. However, the researchers need to be aware of the advantages and disadvantages of these approaches and make their decisions accordingly. Furthermore, we discuss and provide some specific insight requirements for each strategy in order to handle the implementation problems. For this purpose, we review five algorithms and provide a comparison of these algorithms in terms of their affection on prediction accuracy in stock price prediction problem. Adaptive neuro-fuzzy inference system (ANFIS) is used as a model to predict the stock price of a large automotive company available in the Tehran Stock Exchange market. Note that each strategy utilized in the earlier steps of structure identification of ANFIS leads to the necessity of developing different steps for the remaining of the algorithm. Therefore, the performance calculations need not be conclusive since everything else does not stay the same. However, such a comparison might provide some insights about the predictive performance of these methods and their applicability to the stock price prediction problem.

The rest of the paper is organized as follows: Section 2 provides more details on designing a neuro-fuzzy system. Section 3 summarizes the five input selection algorithms which used in the analysis. Section 4 conducts experimental analysis based on stock price prediction problem. In this section, we analyze the results and discuss the advantages and the disadvantages of the algorithms. Finally, Section 5 concludes the paper with our final remarks.

Section snippets

Neuro-fuzzy system modeling

The Takagi–Sugeno–Kang (TSK) system is a fuzzy system with crisp functions in consequents, which perceived convenient for complex applications. TSK systems are widely used in the form of a neuro-fuzzy system called ANFIS. An ANFIS is a fuzzy inference system that can be trained to model some collection of input–output data. The training module allows the system to tune its parameters to learn the input–output relationships hidden in the data set. The ANFIS is composed of two approaches: neural

Input selection algorithms

Many different input selection methods have been proposed in the literature. In this research, our focus is those fundamental ones which have been widely used in non-linear systems specially neuro-fuzzy system modeling. Because of the limited space, the major points of five most popular input selection algorithms are provided here. For more information, readers are referred to the source papers. Three model-based methods and two model-free methods are discussed in the following sub-sections.

Data

Expert systems and evolutionary computing had become a popular choice for financial investing applications. Rada (2008) conducted a comprehensive literature review of expert systems which have been applied in this certain area. Among all types of expert systems, neuro-fuzzy systems, specially ANFIS, have been widely used for financial time series prediction problems (Alizadeh et al., 2009, Esfahanipour and Aghamiri, 2010).

In this paper, the performance of the five input selection algorithms is

Conclusion

Neuro-fuzzy System Modeling approaches have been considered as one of the most widely used data mining tools, especially in prediction and control tasks. Using this modeling approach requires the users to apply one input selection algorithm in order to reduce the complexity of the system and avoid the noise data. Many different methods for the structure identification phase of neuro-fuzzy modeling have been proposed in the literature. In view of the substantial number of existing input

References (30)

  • S. Vieira et al.

    Two cooperative ant colonies for feature selection using fuzzy models

    Expert Systems with Applications

    (2010)
  • Alizadeh, M., Rada, R., Ghoshe Balagh, A. K., & Roshanaei, V. (2009). Forecasting exchange rates: A neuro-fuzzy...
  • Chen, M. S. (1999). A comparative study of learning methods in tuning parameters of fuzzy membership functions. In IEEE...
  • Fazel Zarandi, M. H. (1998). Aggregate system analysis for prediction of tardiness and mixed zones of continuous...
  • Gaweda, A. E., Zurada, J. M., & Setiono, R. (2001). Input selection in data-driven fuzzy modeling. In Proceeding of...
  • Cited by (30)

    • Neuro-fuzzy systems in construction engineering and management research

      2020, Automation in Construction
      Citation Excerpt :

      Alizadeh et al. [137] used ANFIS to study input selection algorithms for data-driven NFS modeling efforts. Five widely used input selection algorithms have been compared based on their complexity, noise-handling capability, accuracy, and robustness [137]. The study identified advantages and disadvantages of input selection algorithms that help researchers choose among them depending on the context of the practical problem for NFS modeling.

    • A new hybrid model for forecasting Brent crude oil price

      2020, Energy
      Citation Excerpt :

      ANFIS is one of the popular AI models combining both fuzzy logic and neural network in its structure [48,49]. Since ANFIS uses if-then rules and several input-outputs, its results can be interpreted with more clarity compared with the explication of other structures such as neural network [50]. Also, Barak et al. [51] document the effectivity of ANFIS in model estimation compared with other neuro-fuzzy paradigms.

    • Forecasting energy consumption using ensemble ARIMA-ANFIS hybrid algorithm

      2016, International Journal of Electrical Power and Energy Systems
    • Improved frequency dynamic in isolated hybrid power system using an intelligent method

      2016, International Journal of Electrical Power and Energy Systems
    View all citing articles on Scopus
    View full text