Discriminant component analysis via distance correlation maximization

Highlights

• We propose a dimensionality reduction technique based on distance correlation.

• Our method maximizes the dependency between data samples and target variable.

• Kernel version of our method is also derived for non-linear problems.

• Our approach has a simple and closed-form solution.

• Our approach is computationally efficient.

Abstract

In the following study, an innovative supervised dimensionality reduction technique is proposed. dCor-based Dimensionality Reduction or dDR technique is based on distance correlation; a powerful correlation measure which is applicable to arbitrary-dimensional random variables. By projecting the samples to a lower dimensional space, dDR maximizes the correlation between explanatory and response variables. The proposed dDR algorithm can be easily implemented and it is computationally efficient. Moreover, it has a closed-form and a simple solution which makes it significantly effective in many different applications. In order to apply the proposed technique on non-linear problems, the kernel version of the dDR is also derived. Extensive analyses and empirical experiments across various visualization, classification, and regression tasks indicate that our algorithm is the method of choice; as it offers statistically superior results in comparison with other state-of-the-art approaches in the literature.

Introduction

Recently, along with the fast pace of developments in technology, science, and huge amount of data available, the need for more robust and efficient learning algorithms has never been felt more. Among these huge data, the existence of high dimensional data sets is a prevalent and inevitable issue.

In high dimensional feature space, due to the curse of dimensionality, the conventional learning algorithms do not produce satisfactory results. In order to maintain a given sample density in many approaches, the number of required training instances and complexity of the target function grows exponentially with increasing data dimension. This situation would become worse even when the dimension of data significantly exceeds the number of data points or the required training data is expensive or difficult to collect. For example, DNA micro-array data consist of thousands of gene expression features while the number of examples is relatively small.

Reducing the data dimensionality has become very popular over the years and many effective methods have been proposed in the literature [15], [20], [38], [48]. From among these techniques, linear dimension reduction methods, which are very prevalent in the literature, try to learn a low-dimensional subspace to project the high dimensional data.

Principle Component Analysis (PCA) [23], an unsupervised linear approach of dimensionality reduction, is a useful statistical technique which has been used in many applications [53]. PCA models the data $X\in {\mathbb{R}}^d$ as approximately lying in some low dimensional subspace. By modeling this subspace, PCA preserves as much variability as possible in the original data.

On the other hand, supervised learning techniques [5], [13], [36] try to predict the l dimensional response variable $Y\in {\mathbb{R}}^l$ from a given explanatory random variable X in order to improve the prediction accuracy of classification or regression tasks.

Significant improvements in learning algorithms are the direct result of the reduction in dimensionality of the input data before training; dimensionality reduction methods are effective tools in overcoming the curse of dimensionality. The performance of supervised learning tasks can be enhanced by considering the discriminative information in response variable while projecting the data into a lower dimensional space. This can be achieved by projecting the data in the direction towards which the dependency between explanatory and response variables is maximized. It is also desirable in dimensionality reduction to preserve the structural information of the data and maximally associate the embedded data with available side information (e.g labels). However, most of these algorithms suffer from very high costs in time and memory or they solve a very complicated optimization problem [14], [41], [44].

In this paper, we propose a supervised dimensionality reduction technique called dCor-based Dimensionality Reduction (dDR). This technique finds a direction to project the data so that it is highly related to response variable Y. In other words, our technique projects the samples into a lower dimensional space while it maximizes the dependency between explanatory random variable X and the target variable Y.

The main contributions of this paper are as follows:

• Our proposed technique can be solved in closed-form efficiently and it does not suffer from high computational complexity or a complicated optimization problem. dDR not only improves the learning ability of learning algorithms, but it also solves the high computational overload of the state-of-the-art approaches. Our proposed method is based on distance correlation, which is more general yet powerful than the Pearson correlation coefficient.

• The characteristics of dDR allow us to derive the kernel version of the algorithm in order to make dDR applicable to non-linear learning problems, as well. These dDR and KdDR algorithms are reduced to a simple optimization problem, which can be solved by eigenvalue decomposition.

• To indicate the effectiveness of our proposed technique, well-known and state-of-the-art dimensionality reduction methods are implemented and compared with our algorithm. Comprehensive analyses and experiments, including time complexity analyses across wide varieties of synthetic, UCI [11], and high dimensional biological data sets over classification and regression problems, are conducted. Our results indicate the effectiveness and efficiency of our approach in processing high dimensional and complex non-linear data structures.

The remainder of the paper is structured as follows: Section 2 introduces the prevalent research progresses in supervised dimensionality reduction in literature. Section 3 describes the distance correlation measure (dCor) on which our method relies on. Section 4 explains our method in a more detailed fashion. Analyses of the results and experimental settings are discussed in Section 5. The concluding Section 6 addresses the conclusion of the paper and our future work.