Data and Dimensionality Reduction in Data Analysis and System Modeling

Definition of the Subject

Data and dimensionality reduction are fundamental pursuits of data analysis and system modeling. With the rapid growth of sizes of data sets and the diversity of data themselves, the use of some reduction mechanisms becomes a necessity. Data reduction is concerned with a reduction of sizes of data sets in terms of the number of data points. This helps reveal an underlying structure in data by presenting a collection of groups present in data. Given a number of groups which is very limited, the clustering mechanisms become effective in terms of data reduction. Dimensionality reduction is aimed at the reduction of the number of attributes (features) of the data which leads to a typically small subset of features or brings the data from a highly dimensional feature space to a new one of a far lower dimensionality. A joint reduction process involves data and feature reduction.

Introduction

In the information age, we are continuously flooded by enormous amounts of...

Appendix: Particle Swarm Optimization (PSO)

In the studies on dimensionality and data reduction, we consider the use of Particle Swarm Optimization (PSO) . PSO is an example of a biologically‐inspired and population‐driven optimization. Originally, the algorithm was introduced by Kennedy and Eberhart [12] where the authors strongly emphasized an inspiration coming from the swarming behavior of animals as well as some inspiration coming from the as well as human social behavior, cf. also [19]. In essence, a particle swarm is a population of particles – possible solutions in the multidimensional search space. Each particle explores the search space and during this search adheres to some quite intuitively appealing guidelines navigating the search process: (a) it tries to follow its previous direction, and (b) it looks back at the best performance both at the level of the individual particle and the entire population. In this sense there is some collective search of the problem space along with some component of memory incorporated as an integral part of the search mechanism.

The performance of each particle during its movement is assessed by means of some performance index. A position of a swarm in the search space, is described by some vector ${ \boldsymbol{z}(t) }$ where “t” denotes consecutive discrete time moments. The next position of the particle is governed by the following update expressions concerning the particle, ${ \boldsymbol{z}(t+1) }$ and its speed, ${ \boldsymbol{v}(t+1) }$

$$ \begin{aligned} & \boldsymbol{z}(t+1) = \boldsymbol{z}(t) + \boldsymbol{v}(t+1)\\ & \qquad\qquad\qquad\quad \text{// update of position of the particle}\\ & \boldsymbol{v}(t+1) = \xi \boldsymbol{v}(t) + \phi_{1}(\boldsymbol{p}- \boldsymbol{x}(t)) + \phi_{2} (\boldsymbol{p}_\mathrm{total}-\boldsymbol{x}(t))\\ & \qquad\qquad\qquad\quad \text{// update of speed of the particle} \end{aligned} $$

where ${ \boldsymbol{p} }$ denotes the best position (the lowest performance index) reported so far for this particle, ${ \boldsymbol{p}_\text{total} }$ is the best position overall developed so far across the whole population. ϕ₁ and ϕ₂ are random number drawn from the uniform distribution ${ U[0,2] }$ that help build a proper mix of the components of the speed; different random numbers affect the individual coordinates of the speed. The second expression governing the change in the velocity of the particle is particularly interesting as it nicely captures the relationships between the particle and its history as well as the history of the overall population in terms of their performance reported so far.

There are three components determining the updated speed of the particle. First, the current speed ${ \boldsymbol{v}(t) }$ is scaled by the inertial weight (ξ) smaller than 1 whose role is to articulate some tendency to a drastic change of the current speed. Second, we relate to the memory of this particle by recalling the best position of the particle achieved so far. Thirdly, there is some reliance on the best performance reported across the whole population (which is captured by the last component of the expression governing the speed adjustment).