Skip to main content
SpringerLink
Log in

Kernel feature extraction methods observed from the viewpoint of generating-kernels

  • Research Article
  • Published:
Frontiers of Electrical and Electronic Engineering in China

Abstract

This paper introduces an idea of generating a kernel from an arbitrary function by embedding the training samples into the function. Based on this idea, we present two nonlinear feature extraction methods: generating kernel principal component analysis (GKPCA) and generating kernel Fisher discriminant (GKFD). These two methods are shown to be equivalent to the function-mapping-space PCA (FMS-PCA) and the function-mapping-space linear discriminant analysis (FMS-LDA) methods, respectively. This equivalence reveals that the generating kernel is actually determined by the corresponding function map. From the generating kernel point of view, we can classify the current kernel Fisher discriminant (KFD) algorithms into two categories: KPCA + LDA based algorithms and straightforward KFD (SKFD) algorithms. The KPCA + LDA based algorithms directly work on the given kernel and are not suitable for non-kernel functions, while the SKFD algorithms essentially work on the generating kernel from a given symmetric function and are therefore suitable for non-kernels as well as kernels. Finally, we outline the tensor-based feature extraction methods and discuss ways of extending tensor-based methods to their generating kernel versions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Vapnik V. The Nature of Statistical Learning Theory. New York: Springer, 1995

    MATH  Google Scholar 

  2. Müller K R, Mika S, Rätsch G, Tsuda K, Schölkopf B. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 2001, 12(2): 181–201

    Article  Google Scholar 

  3. Schölkopf B, Smola A, Muller K R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 1998, 10(5): 1299–1319

    Article  Google Scholar 

  4. Mika S, Rätsch G, Weston J, Schölkopf B, Müller K R. Fisher discriminant analysis with kernels. In: Proceedings of IEEE International Workshop on Neural Networks for Signal Processing IX. 1999, 41–48

  5. Mika S, Rätsch G, Schölkopf B, Smola A, Weston J, Müller K R. Invariant feature extraction and classification in kernel spaces. Advances in Neural Information Processing Systems, 1999, 12: 526–532

    Google Scholar 

  6. Baudat G, Anouar F. Generalized discriminant analysis using a kernel approach. Neural Computation, 2000, 12(10): 2385–2404

    Article  Google Scholar 

  7. Roth V, Steinhage V. Nonlinear discriminant analysis using kernel functions. In: Solla S A, Leen T K, Mueller K R, eds. Advances in Neural Information Processing Systems. 2000, 12: 568–574

  8. Mika S, Rätsch G, Weston J, Schölkopf B, Smola A, Müller K R. Constructing descriptive and discriminative non-linear features: Rayleigh coefficients in kernel feature spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(5): 623–628

    Article  Google Scholar 

  9. Yang M H. Kernel Eigenfaces vs. kernel Fisherfaces: face recognition using kernel methods. In: Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition. 2002, 215–220

  10. Lu J, Plataniotis K N, Venetsanopoulos A N. Face recognition using kernel direct discriminant analysis algorithms. IEEE Transactions on Neural Networks, 2003, 14(1): 117–126

    Article  Google Scholar 

  11. Schölkopf B, Smola A. Learning with Kernels. Cambridge: MIT Press, 2002

    Google Scholar 

  12. Shawe-Taylor J, Cristianini N. Kernel Methods for Pattern Analysis. Cambridge: Cambridge University Press, 2004

    Google Scholar 

  13. Xu J, Zhang X, Li Y. Kernel MSE algorithm: a unified framework for KFD, LS-SVM, and KRR. In: Proceedings of the International Joint Conference on Neural Networks. 2001, 1486–1491

  14. Billings S A, Lee K L. Nonlinear fisher discriminant analysis using a minimum squared error cost function and the orthogonal least squares algorithm. Neural Networks, 2002, 15(2): 263–270

    Article  Google Scholar 

  15. Zheng W, Zhao L, Zou C. Foley-Sammon optimal discriminant vectors using kernel approach. IEEE Transactions on Neural Networks, 2005, 16(1): 1–9

    Article  Google Scholar 

  16. Yang J, Jin Z, Yang J Y, Zhang D, Frangi A F. Essence of kernel Fisher discriminant: KPCA plus LDA. Pattern Recognition, 2004, 37(10): 2097–2100

    Article  Google Scholar 

  17. Yang J, Frangi A F, Yang J Y, Zhang D, Jin Z. KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(2): 230–244

    Article  Google Scholar 

  18. Xu Y, Zhang D, Jin Z, Li M, Yang J Y. A fast kernel-based nonlinear discriminant analysis for multi-class problems. Pattern Recognition, 2006, 39(6): 1026–1033

    Article  MATH  Google Scholar 

  19. Zhao J, Wang H, Ren H, Kee S C. LBP discriminant analysis for face verification. In: Proceeding of the 2005 IEEE Conference on Computer Vision and Pattern Recognition. 2005, 3: 167

    Google Scholar 

  20. Liu C. Capitalize on dimensionality increasing techniques for improving face recognition grand challenge performance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(5): 725–737

    Article  Google Scholar 

  21. Cevikalp H, Neamtu M, Wilkes M, Barkana A. Discriminative common vector method with kernels. IEEE Transactions on Neural Networks, 2006, 17(6): 1550–1565

    Article  Google Scholar 

  22. Bach F R, Jordan MI. Kernel independent component analysis. Journal of Machine Learning Research, 2002, 3(1): 1–48

    Article  MathSciNet  Google Scholar 

  23. Yang J, Gao X, Zhang D, Yang J Y. Kernel ICA: an alternative formulation and its application to face recognition. Pattern Recognition, 2005, 38(10): 1784–1787

    Article  MATH  Google Scholar 

  24. Ma J. Function replacement vs. kernel trick. Neurocomputing, 2003, 50: 479–483

    Article  MATH  Google Scholar 

  25. Ma J, Theiler J, Perkins S. Two realizations of a general feature extraction framework. Pattern Recognition, 2004, 37(5): 875–887

    Article  Google Scholar 

  26. Lodhi H, Saunders C, Shawe-Taylor J, Cristianini N, Watkins C. Text classification using string kernels. Journal of Machine Learning Research, 2002, 2(2): 419–444

    Article  MATH  Google Scholar 

  27. Chen W S, Yuen P C, Huang J, Lai J. Wavelet kernel construction for kernel discriminant analysis on face recognition. In: Proceeding of the 2006 Conference on Computer Vision and Pattern Recognition Workshop. 2006, 47–52

  28. Schölkopf B. Support vector learning. Dissertation for the Doctoral Degree. Berlin: Berlin Technical University, 1997

    Google Scholar 

  29. Camps-Valls G, Martin-Guerrero J, Rojo-Alvarez J, Soria-Olivas E. Fuzzy sigmoid kernel for support vector classifiers. Neurocomputing, 2004, 62: 501–506

    Article  Google Scholar 

  30. Ahonen T, Hadid A, Pietikäinen M. Face description with local binary patterns: application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(12): 2037–2041

    Article  Google Scholar 

  31. Mangasarian L. Generalized support vector machines. In: Smola A J, Bartlett P, Schokopf B, Schuurmans D, eds. Advances in Large Margin Classifiers. 2000, 135–146

  32. Golub G H, Van Loan C F. Matrix Computations. 3rd ed. Baltimore: The Johns Hopkins University Press, 1996

    MATH  Google Scholar 

  33. Swets D L, Weng J. Using discriminant eigenfeatures for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 18(8): 831–836

    Article  Google Scholar 

  34. Belhumeur P N, Hespanha J P, Kriengman D J. Eigenfaces vs. Fisherfaces: Recognition using class specific linear pro jection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 711–720

    Article  Google Scholar 

  35. Yang J, Yang J Y. Why can LDA be performed in PCA transformed space? Pattern Recognition, 2003, 36(2): 563–566

    Article  Google Scholar 

  36. Liu C J, Wechsler H. Robust coding schemes for indexing and retrieval from large face databases. IEEE Transactions on Image Processing, 2000, 9(1): 132–137

    Article  Google Scholar 

  37. Turk M, Pentland A. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 1991, 3(1): 71–86

    Article  Google Scholar 

  38. Liu K, Cheng Y Q, Yang J Y. Algebraic feature extraction for image recognition based on an optimal discriminant criterion. Pattern Recognition, 1993, 26(6): 903–911

    Article  Google Scholar 

  39. Yang J, Zhang D, Frangi A F, Yang J Y. Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 131–137

    Article  Google Scholar 

  40. Visani M, Garcia C, Laurent C. Comparing robustness of two-dimensional PCA and eigenfaces for face recognition. Lecture Notes in Computer Science, 2004, 3212: 717–724

    Article  Google Scholar 

  41. Zuo W M, Zhang D. K. Wang K. An assembled matrix distance metric for 2DPCA-based image recognition. Pattern Recognition Letters, 2006, 27(3): 210–216

    Google Scholar 

  42. Zhang D Q, Zhou Z H. (2D)(2)PCA: Two-directional two-dimensional PCA for efficient face representation and recognition. Neurocomputing, 2005, 69(1–3): 224–231

    Article  Google Scholar 

  43. Kong H, Wang L, Teoh E K, Li X, Wang J G, Venkateswarlu R. Generalized 2D principal component analysis for face image representation and recognition. Neural Networks, 2005, 18(5–6): 585–594

    Article  Google Scholar 

  44. Ye J. Generalized low rank approximations of matrices. Machine Learning, 2005, 61(1–3): 167–191

    Article  MATH  Google Scholar 

  45. Hyvärinen A, Oja E. Independent component analysis: algorithms and applications. Neural Networks, 2000, 13(4–5): 411–430

    Article  Google Scholar 

  46. He X, Yan S, Hu Y, Niyogi P, Zhang H J. Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(3): 328–340

    Article  Google Scholar 

  47. Ye J, Janardan R, Li Q. Two-dimensional linear discriminant analysis. Advances in Neural Information Processing Systems, 2004, 17: 1569–1576

    Google Scholar 

  48. Li M, Yuan B. 2D-LDA: a novel statistical linear discriminant analysis for image matrix. Pattern Recognition Letters, 2005, 26(5): 527–532

    Article  Google Scholar 

  49. Xiong H, Swamy M, Ahmad M. Two-dimensional FLD for face recognition. Pattern Recognition, 2005, 38(7): 1121–1124

    Article  Google Scholar 

  50. Yang J, Zhang D, Yong X, Yang J. Two-dimensional linear discriminant transform for face recognition. Pattern Recognition, 2005, 38(7): 1125–1129

    Article  MATH  Google Scholar 

  51. Zuo W, Zhang D, Yang J, Wang K. BDPCA plus LDA: A novel fast feature extraction technique for face recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 2006, 36(4): 946–953

    Article  Google Scholar 

  52. Zou C, Sun N, Ji Z, Zhao L. 2DCCA: a novel method for small sample size face recognition. In: Proceedings of IEEE Workshop on Applications of Computer Vision. 2007, 43

  53. Lee S H, Choi S. Two-dimensional canonical correlation analysis. IEEE Signal Processing Letters, 2007, 14(10): 735–738

    Article  Google Scholar 

  54. Gao Q, Zhang L, Zhang D, Xu H. Independent components extraction from image matrix. Pattern Recognition Letters, 2010, 31(3): 171–178

    Article  Google Scholar 

  55. Chen S B, Zhao H F, Kong M, Luo B. 2d-lpp: a two-dimensional extension of locality preserving projections. Neurocomputing, 2007, 70(4–6): 912–921

    Google Scholar 

  56. Hu D W, Feng G Y, Zhou Z T. Two-dimensional locality preserving projections (2DLPP) with its application to palmprint recognition. Pattern Recognition, 2007, 40(1): 339–342

    Article  MATH  Google Scholar 

  57. Xu D, Yan S, Zhang L, Lin S, Zhang H J, Huang T S. Reconstruction and recognition of tensor-based objects with concurrent subspaces analysis. IEEE Transactions on Circuits and Systems for Video Technology, 2008, 18(1): 36–47

    Article  Google Scholar 

  58. Wang H, Ahuja N. A tensor approximation approach to dimensionality reduction. Journal of Computer Vision, 2008, 76(3): 217–229

    Article  Google Scholar 

  59. Tao D, Li X, Wu X, Maybank S J. General tensor discriminant analysis and gabor features for gait recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(10): 1700–1715

    Article  Google Scholar 

  60. Lu H, Plataniotis K N, Venetsanopoulos A N. MPCA: multilinear principal component analysis of tensor objects. IEEE Transactions on Neural Networks, 2008, 19(1): 18–39

    Article  Google Scholar 

  61. Zhang L, Gao Q, Zhang D. Directional independent component analysis with tensor representation. In: Proceedings of Computer Vision and Pattern Recognition. 2008, 1–7

  62. Sun N, Wang H, Ji Z, Zou C, Zhao L. An efficient algorithm for kernel two-dimensional principal component analysis. Neural Computing & Applications, 2008, 17(1): 59–64

    Google Scholar 

  63. Liu J, Chen S, Zhou Z H, Tan X. Generalized low-rank approximations of matrices revisited. IEEE Transactions on Neural Networks, 2010, 21(4): 621–632

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing, 210094, China

    Jian Yang

Authors
  1. Jian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Yang.

Additional information

Jian YANG received the BS degree in mathematics from the Xuzhou Normal University in 1995. He received the MS degree in applied mathematics from the Changsha Railway University in 1998 and the PhD degree from the Nanjing University of Science and Technology (NUST), on the subject of pattern recognition and intelligence systems in 2002. In 2003, he was a postdoctoral researcher at the University of Zaragoza, and in the same year, he was awarded the RyC program Research Fellowship sponsored by the Spanish Ministry of Science and Technology. From 2004 to 2006, he was a Postdoctoral Fellow at Biometrics Centre of Hong Kong Polytechnic University. From 2006 to 2007, he was a Postdoctoral Fellow at Department of Computer Science of New Jersey Institute of Technology. Now, he is a professor in the School of Computer Science and Technology of NUST. He is the author of more than 50 scientific papers in pattern recognition and computer vision. His journal papers have been cited more than 1000 times in the ISI Web of Science, and 2000 times in the Web of Scholar Google. His research interests include pattern recognition, computer vision and machine learning. Currently, he is an associate editor of Pattern Recognition Letters and Neurocomputing, respectively.

About this article

Cite this article

Yang, J. Kernel feature extraction methods observed from the viewpoint of generating-kernels. Front. Electr. Electron. Eng. China 6, 43–55 (2011). https://doi.org/10.1007/s11460-011-0129-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11460-011-0129-z

Keywords

Search

Navigation