Elsevier

Pattern Recognition

Volume 44, Issue 7, July 2011, Pages 1540-1551
Pattern Recognition

A survey of multilinear subspace learning for tensor data

https://doi.org/10.1016/j.patcog.2011.01.004Get rights and content

Abstract

Increasingly large amount of multidimensional data are being generated on a daily basis in many applications. This leads to a strong demand for learning algorithms to extract useful information from these massive data. This paper surveys the field of multilinear subspace learning (MSL) for dimensionality reduction of multidimensional data directly from their tensorial representations. It discusses the central issues of MSL, including establishing the foundations of the field via multilinear projections, formulating a unifying MSL framework for systematic treatment of the problem, examining the algorithmic aspects of typical MSL solutions, and categorizing both unsupervised and supervised MSL algorithms into taxonomies. Lastly, the paper summarizes a wide range of MSL applications and concludes with perspectives on future research directions.

Introduction

With the advances in data collection and storage capabilities, massive multidimensional data are being generated on a daily basis in a wide range of emerging applications, and learning algorithms for knowledge extraction from these data are becoming more and more important. Two-dimensional (2D) data include gray-level images in computer vision and pattern recognition [1], [2], [3], [4], multichannel EEG signals in biomedical engineering [5], [6], and gene expression data in bioinformatics [7]. Three-dimensional (3D) data include 3D objects in generic object recognition [8], hyperspectral cube in remote sensing [9], and gray-level video sequences in activity or gesture recognition for surveillance or human–computer interaction (HCI) [10], [11]. There are also many multidimensional signals in medical image analysis [12], content-based retrieval [1], [13], and space-time super-resolution [14] for digital cameras with limited spatial and temporal resolution. In addition, many streaming data and mining data are frequently organized as third-order tensors [15], [16], [17]. Data in environmental sensor monitoring are often organized in three modes of time, location, and type [17]. Data in social network analysis are usually organized in three modes of time, author, and keywords [17]. Data in network forensics are often organized in three modes of time, source, and destination, and data in web graph mining are commonly organized in three modes of source, destination, and text [15].

These massive multidimensional data are usually very high-dimensional, with a large amount of redundancy, and only occupying a subspace of the input space [18]. Thus, for feature extraction, dimensionality reduction is frequently employed to map high-dimensional data to a low-dimensional space while retaining as much information as possible [18], [19]. However, this is a challenging problem due to the large variability and complex pattern distribution of the input data, and the limited number of samples available for training in practice [20]. Linear subspace learning (LSL) algorithms are traditional dimensionality reduction techniques that represent input data as vectors and solve for an optimal linear mapping to a lower-dimensional space. Unfortunately, they often become inadequate when dealing with massive multidimensional data. They result in very high-dimensional vectors, lead to the estimation of a large number of parameters, and also break the natural structure and correlation in the original data [21], [2], [22].

Due to the challenges in emerging applications above, there has been a pressing need for more effective dimensionality reduction schemes for massive multidimensional data. Recently, interests have grown in multilinear subspace learning (MSL) [2], [21], [22], [23], [24], [25], [26], a novel approach to dimensionality reduction of multidimensional data where the input data are represented in their natural multidimensional form as tensors. Fig. 1 shows two examples of tensor data representations for a face image and a silhouette sequence. MSL has the potential to learn more compact and useful representations than LSL [21], [27] and it is expected to have potential future impact in both developing new MSL algorithms and solving problems in applications involving massive multidimensional data. The research on MSL has gradually progressed from heuristic exploration to systematic investigation [28] and both unsupervised and supervised MSL algorithms have been proposed in the past a few years [2], [21], [22], [23], [24], [25], [26].

It should be noted that MSL belongs to tensor data analysis (or tensor-based computation and modeling), which is more general and has a much wider scope. Multilinear algebra, the basis of tensor data analysis, has been studied in mathematics for several decades [29], [30], [31] and there are a number of recent survey papers summarizing recent developments in tensor data analysis. E.g., Qi et al. review numerical multilinear algebra and its applications in [32]. Muti and Bourennane [33] survey new filtering methods for multicomponent data modeled as tensors in noise reduction for color images and multicomponent seismic data. Acar and Yener [34] survey unsupervised multi-way data analysis for discovering structures in higher-order data sets in applications such as chemistry, neuroscience, and social network analysis. Kolda and Bader [35] provide an overview of higher-order tensor decompositions and their applications in psychometrics, chemometrics, signal processing, etc. These survey papers primarily focus on unsupervised tensor data analysis through factor decomposition. In addition, Zafeiriou [36] provides an overview of both unsupervised and supervised nonnegative tensor factorization (NTF) [37], [38] with NTF algorithms and their applications in visual representation and recognition discussed.

In contrast, this survey paper focuses on a systematic introduction to the field of MSL. To the best knowledge of the authors, this is the first unifying survey of both unsupervised and supervised MSL algorithms. For detailed introduction and review on multilinear algebra, multilinear decomposition, and NTF, the readers should go to [30], [31], [32], [33], [34], [35], [36], [39], [40] while this paper serves as a complement to these surveys. In the rest of this paper, Section 2 first introduces notations and basic multilinear algebra, and then addresses multilinear projections for direct mapping from high-dimensional tensorial representations to low-dimensional vectorial or tensorial representations. This section also provides insights to the relationships among different projections. Section 3 formulates a unifying MSL framework for systematic treatment of the MSL problem. A typical approach to solve MSL is presented and several algorithmic issues are examined in this section. Under the MSL framework, existing MSL algorithms are reviewed, analyzed and categorized into taxonomies in Section 4. Finally, MSL applications are summarized in Section 5 and future research topics are covered in Section 6. For easy reference, Table 1 lists several important acronyms used in this paper.

Section snippets

Fundamentals and multilinear projections

This section first reviews the notations and some basic multilinear operations [30], [31], [41] that are necessary in defining the MSL problem. The important concepts of multilinear projections are then introduced, including elementary multilinear projection (EMP), tensor-to-vector projection (TVP), and tensor-to-tensor projection (TTP), and their relationships are explored. Table 2 summarizes the important symbols used in this paper for quick reference.

The multilinear subspace learning framework

This section formulates a general MSL framework. It defines the MSL problem in a similar way as LSL, as well as tensor and scalar scatter measures for optimality criterion construction. It also outlines a typical solution and discusses related issues.

Taxonomy of multilinear subspace learning algorithms

This section reviews several important MSL algorithms under the MSL framework. Due to the fundamentality and importance of PCA and LDA, the focus is on the multilinear extensions of these two classical linear algorithms. Fig. 8, Fig. 9 depict taxonomies for these algorithms, one for multilinear extensions of PCA and the other for multilinear extensions of LDA, respectively, which will be discussed in detail in the following. The historical basis of MSL will also be described.

Multilinear subspace learning applications

Due to the advances in sensor and storage technology, MSL is becoming increasingly popular in a wide range of application domains involving tensor-structured data sets. This section will summarize several applications of MSL algorithms in real-world applications, including face recognition and gait recognition in biometrics, music genre classification in audio signal processing, EEG signal classification in biomedical engineering, anomaly detection in data mining, and visual content analysis in

Conclusions and future works

This paper presents a survey of an emerging dimensionality reduction approach for direct feature extraction from tensor data: multilinear subspace learning. It reduces the dimensionality of massive data directly from their natural multidimensional representation: tensors. This survey covers multilinear projections, MSL framework, typical MSL solutions, MSL taxonomies, and MSL applications. MSL is a new field with many open issues to be examined further. The rest of this section outlines several

Acknowledgment

The authors would like to thank the anonymous reviewers for their insightful comments, which have helped to improve the quality of this paper.

Haiping Lu received the B.Eng. and M.Eng. degrees in electrical and electronic engineering from Nanyang Technological University, Singapore, in 2001 and 2004, respectively, and the Ph.D. degree in electrical and computer engineering from University of Toronto, Toronto, ON, Canada, in 2008. Currently, he is a research fellow in the Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore. His current research interests include pattern recognition, machine learning,

References (113)

  • J. Li et al.

    A prior neurophysiologic knowledge free tensor-based scheme for single trial eeg classification

    IEEE Transactions on Neural Systems and Rehabilitation Engineering

    (2009)
  • H. Lu, K.N. Plataniotis, A.N. Venetsanopoulos, Regularized common spatial patterns with generic learning for EEG signal...
  • J. Ye et al.

    Using uncorrelated discriminant analysis for tissue classification with gene expression data

    IEEE/ACM Transactions on Computational Biology Bioinformatics

    (2004)
  • H.S. Sahambi et al.

    A neural-network appearance-based 3-D object recognition using independent component analysis

    IEEE Transactions on Neural Networks

    (2003)
  • N. Renard et al.

    Dimensionality reduction based on tensor modeling for classification methods

    IEEE Transactions on Geoscience and Remote Sensing

    (2009)
  • R. Chellappa et al.

    Recognition of Humans and their Activities using Video

    (2005)
  • R.D. Green et al.

    Quantifying and recognizing human movement patterns from monocular video images-part II: applications to biometrics

    IEEE Transactions on Circuits and Systems for Video Technology

    (2004)
  • D.R. Hardoon et al.

    Decomposing the tensor kernel support vector machine for neuroscience data with structure labels

    Machine Learning

    (2010)
  • X. He, Incremental semi-supervised subspace learning for image retrieval, in: ACM Conference on Multimedia 2004, 2004,...
  • E. Shechtman et al.

    Space-time super-resolution

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2005)
  • J. Sun, D. Tao, C. Faloutsos, Beyond streams and graphs: dynamic tensor analysis, in: Proceedings the 12th ACM SIGKDD...
  • J. Sun et al.

    Less is more: sparse graph mining with compact matrix decomposition

    Statistical Analysis and Data Mining

    (2008)
  • J. Sun et al.

    Incremental tensor analysis: theory and applications

    ACM Transactions on Knowledge Discovery from Data

    (2008)
  • G. Shakhnarovich et al.

    Face recognition in subspaces

  • I.T. Jolliffe

    Principal Component Analysis

    (2002)
  • S.Z. Li et al.

    Introduction

  • H. Lu et al.

    MPCA: multilinear principal component analysis of tensor objects

    IEEE Transactions on Neural Networks

    (2008)
  • D. Tao et al.

    Supervised tensor learning

    Knowledge and Information Systems

    (2007)
  • D. Tao et al.

    General tensor discriminant analysis and gabor features for gait recognition

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2007)
  • H. Lu et al.

    Uncorrelated multilinear principal component analysis for unsupervised multilinear subspace learning

    IEEE Transactions on Neural Networks

    (2009)
  • H. Lu et al.

    Uncorrelated multilinear discriminant analysis with regularization and aggregation for tensor object recognition

    IEEE Transactions on Neural Networks

    (2009)
  • D. Xu et al.

    Reconstruction and recognition of tensor-based objects with concurrent subspaces analysis

    IEEE Transactions on Circuits and Systems for Video Technology

    (2008)
  • H. Lu et al.

    A taxonomy of emerging multilinear discriminant analysis solutions for biometric signal recognition

  • W.H. Greub

    Multilinear Algebra

    (1967)
  • L.D. Lathauwer et al.

    A multilinear singular value decomposition

    SIAM Journal of Matrix Analysis and Applications

    (2000)
  • L.D. Lathauwer et al.

    On the best rank-1 and rank-(R1,R2,…,RN) approximation of higher-order tensors

    SIAM Journal of Matrix Analysis and Applications

    (2000)
  • L. Qi et al.

    Numerical multilinear algebra and its applications

    Frontiers of Mathematics in China

    (2007)
  • E. Acar et al.

    Unsupervised multiway data analysis: a literature survey

    IEEE Transactions on Knowledge and Data Engineering

    (2009)
  • T.G. Kolda et al.

    Tensor decompositions and applications

    SIAM Review

    (2009)
  • S. Zafeiriou

    Algorithms for nonnegative tensor factorization

  • T. Hazan, S. Polak, A. Shashua, Sparse image coding using a 3D non-negative tensor factorization, in: Proceedings of...
  • A. Shashua, T. Hazan, Non-negative tensor factorization with applications to statistics and computer vision, in:...
  • R. Bro, Multi-way analysis in the food industry – models, algorithms and applications, Ph.D. Thesis, University of...
  • A.K. Smilde et al.

    Multi-way Analysis

    (2004)
  • B.W. Bader et al.

    Algorithm 862: Matlab tensor classes for fast algorithm prototyping

    ACM Transactions on Mathematical Software

    (2006)
  • H. Lu, Multilinear subspace learning for face and gait recognition, Ph.D. Thesis, University of Toronto, 2008. URL...
  • X. He, D. Cai, P. Niyogi, Tensor subspace analysis, in: Advances in Neural Information Processing Systems 18 (NIPS),...
  • R.O. Duda et al.

    Pattern Classification

    (2001)
  • T.K. Moon et al.

    Mathematical Methods and Algorithms for Signal Processing

    (2000)
  • Y. Wang, S. Gong, Tensor discriminant analysis for view-based object recognition, in: Proceedings of International...
  • Cited by (328)

    View all citing articles on Scopus

    Haiping Lu received the B.Eng. and M.Eng. degrees in electrical and electronic engineering from Nanyang Technological University, Singapore, in 2001 and 2004, respectively, and the Ph.D. degree in electrical and computer engineering from University of Toronto, Toronto, ON, Canada, in 2008. Currently, he is a research fellow in the Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore. His current research interests include pattern recognition, machine learning, biometrics, and biomedical engineering.

    Konstantinos N. Plataniotis is a Professor with the Department of Electrical and Computer Engineering and the Director of the Knowledge Media Design Institute at the University of Toronto. He received his B.Eng. degree in Computer Engineering from University of Patras, Greece in 1988 and his M.S. and Ph.D. degrees in Electrical Engineering from Florida Institute of Technology in 1992 and 1994, respectively. His research interests include multimedia systems, biometrics, image and signal processing, communications systems and pattern recognition. He is a registered professional engineer in Ontario, and the Editor-in-Chief (2009–2011) for the IEEE Signal Processing Letters.

    Anastasios N. Venetsanopoulos is a Professor of Electrical and Computer Engineering at Ryerson University, Toronto, and a Professor Emeritus with the Department of Electrical and Computer Engineering at the University of Toronto. He received the B.Eng. degree in electrical and mechanical engineering from the National Technical University of Athens, Greece in 1965, and the M.S., M.Phil., and Ph.D. degrees in Electrical Engineering from Yale University in 1966, 1968 and 1969, respectively. His research interests include multimedia, digital signal/image processing, telecommunications, and biometrics. He is a Fellow of the Engineering Institute of Canada, the IEEE, the Canadian Academy of Engineering, and the Royal Society Of Canada.

    View full text