Skip to main content
SpringerLink
Log in

Regularized linear discriminant analysis based on generalized capped \(l_{2,q}\)-norm

  • Original Research
  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

Aiming to improve the robustness and adaptiveness of the recently investigated capped norm linear discriminant analysis (CLDA), this paper proposes a regularized linear discriminant analysis based on the generalized capped \(l_{2,q}\)-norm (GCLDA). Compared to CLDA, there are two improvements in GCLDA. Firstly, GCLDA uses the capped \(l_{2,q}\)-norm rather than the capped \(l_{2,1}\)-norm to measure the within-class and between-class distances for arbitrary \(q>0\). By selecting an appropriate q, GCLDA is adaptive to different data, and also removes extreme outliers and suppresses the effect of noise more effectively. Secondly, by taking into account a regularization term, GCLDA not only improves its generalization ability but also avoids singularity. GCLDA is solved through a series of generalized eigenvalue problems. Experiments on an artificial dataset, some real world datasets and a high-dimensional dataset demonstrate the effectiveness of GCLDA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997). Eigenfaces versus fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 711–720.

    Article  Google Scholar 

  • Chen, X., Yang, J., & Jin, Z. (2014). An improved linear discriminant analysis with l1-norm for robust feature extraction. In 2014 22nd international conference on pattern recognition (pp. 1585–1590). IEEE.

  • Ding, C., Zhou, D., He, X., Zha, H. (2006), \(r_1\)-pca: Rotational invariant \(l_1\)-norm principal component analysis for robust subspace factorization. In Proceedings of the 23rd international conference on machine learning (pp. 281–288).

  • Filzmoser, P., Maronna, R., & Werner, M. (2008). Outlier identification in high dimensions. Computational Statistics & Data Analysis, 52(3), 1694–1711.

    Article  Google Scholar 

  • Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2), 179–188.

    Article  Google Scholar 

  • Friedman, J. H. (1989). Regularized discriminant analysis. Journal of the American Statistical Association, 84(405), 165–175.

    Article  Google Scholar 

  • Fukunaga, K. (2013). Introduction to statistical pattern recognition. Elsevier.

    Google Scholar 

  • Gao, H., Nie, F., Cai, W., & Huang, H. (2015). Robust capped norm nonnegative matrix factorization: Capped norm nmf. In Proceedings of the 24th ACM international on conference on information and knowledge management (pp. 871–880).

  • Gschwandtner, M., & Filzmoser, P. (2013). Outlier detection in high dimension using regularization. In Synergies of soft computing and statistics for intelligent data analysis (pp. 237–244). Springer.

  • Guo, Y., Hastie, T., & Tibshirani, R. (2007). Regularized linear discriminant analysis and its application in microarrays. Biostatistics, 8(1), 86–100.

    Article  Google Scholar 

  • He, X., & Fung, W. K. (2000). High breakdown estimation for multiple populations with applications to discriminant analysis. Journal of Multivariate Analysis, 72(2), 151–162.

    Article  Google Scholar 

  • Howland, P., Jeon, M., & Park, H. (2003). Structure preserving dimension reduction for clustered text data based on the generalized singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 25(1), 165–179.

    Article  Google Scholar 

  • Huang, R., Liu, C., & Zhou, J. (2017). Discriminant analysis via jointly l\(_{2,1}\)-norm sparse tensor preserving embedding for image classification. Journal of Visual Communication and Image Representation, 47, 10–22.

    Article  Google Scholar 

  • Hubert, M., & Van Driessen, K. (2004). Fast and robust discriminant analysis. Computational Statistics & Data Analysis, 45(2), 301–320.

    Article  Google Scholar 

  • Jureckova, J., & Picek, J. (2005). Robust statistical methods. Chapman and Hall/CRC.

    Book  Google Scholar 

  • Kalina, J., & Tichavskỳ, J. (2021). The minimum weighted covariance determinant estimator for high-dimensional data. Advances in Data Analysis and Classification 1–23.

  • Kepplinger, D. (2015). Discriminant analysis based on robust regularized covariance estimation. PhD thesis.

  • Lai, Z., Liu, N., Shen, L., & Kong, H. (2018). Robust locally discriminant analysis via capped norm. IEEE Access, 7, 4641–4652.

    Article  Google Scholar 

  • Lan, G., Hou, C., & Yi, D. (2016). Robust feature selection via simultaneous capped \(\ell _2\)-norm and \(\ell _{2,1}\)-norm minimization. In 2016 IEEE international conference on big data analysis (ICBDA) (pp 1–5). IEEE.

  • Li, X., Hu, W., Wang, H., & Zhang, Z. (2010). Linear discriminant analysis using rotational invariant l1 norm. Neurocomputing, 73(13–15), 2571–2579.

    Article  Google Scholar 

  • Lin, H. T., Chou, T. Y., Chen, Y. T., & Huang, Y. C. (2014). Profitability analysis using idea-da framework. Annals of Operations Research, 223(1), 291–308.

    Article  Google Scholar 

  • Li, C. N., Shang, M. Q., Shao, Y. H., Xu, Y., Liu, L. M., & Wang, Z. (2019). Sparse l1-norm two dimensional linear discriminant analysis via the generalized elastic net regularization. Neurocomputing, 337, 80–96.

    Article  Google Scholar 

  • Li, C. N., Shao, Y. H., & Deng, N. Y. (2015). Robust l1-norm two-dimensional linear discriminant analysis. Neural Networks, 65, 92–104.

    Article  Google Scholar 

  • Li, C. N., Shao, Y. H., Wang, Z., & Deng, N. Y. (2019). Robust bilateral lp-norm two-dimensional linear discriminant analysis. Information Sciences, 500, 274–297.

    Article  Google Scholar 

  • Li, C. N., Shao, Y. H., Wang, Z., Deng, N. Y., & Yang, Z. M. (2019). Robust bhattacharyya bound linear discriminant analysis through an adaptive algorithm. Knowledge-Based Systems, 183, 104858.

    Article  Google Scholar 

  • Li, C. N., Shao, Y. H., Yin, W., & Liu, M. Z. (2019). Robust and sparse linear discriminant analysis via an alternating direction method of multipliers. IEEE Transactions on Neural Networks and Learning Systems, 31(3), 915–926.

    Article  Google Scholar 

  • Liu, J., Xiong, X., Ren, P. W., Zhao, D., Li, C. N., & Shao, Y. H. (2020). Capped norm linear discriminant analysis and its applications. arXiv preprint arXiv:2011.02147

  • Liu, Y., Gao, Q., Miao, S., Gao, X., Nie, F., & Li, Y. (2016). A non-greedy algorithm for l1-norm lda. IEEE Transactions on Image Processing, 26(2), 684–695.

    Article  Google Scholar 

  • Li, M., Wang, J., Wang, Q., & Gao, Q. (2017). Trace ratio 2dlda with l1-norm optimization. Neurocomputing, 266, 216–225.

    Article  Google Scholar 

  • Li, C. N., Zheng, Z. R., Liu, M. Z., Shao, Y. H., & Chen, W. J. (2017). Robust recursive absolute value inequalities discriminant analysis with sparseness. Neural Networks, 93, 205–218.

    Article  Google Scholar 

  • Ma, X., Zhao, M., Zhang, Z., Fan, J., & Zhan, C. (2018). Anchored projection based capped \(l_{2, 1}\)-norm regression for super-resolution. In Pacific Rim international conference on artificial intelligence (pp. 10–18). Springer.

  • Nene, S. A., Nayar, S. K., & Murase, H. (1996). Columbia object image library (coil-100). Columbia University.

    Google Scholar 

  • Nie, F., Huang, H., Cai, X., & Ding, C. (2010). Efficient and robust feature selection via joint l2,1-norms minimization. Advances in Neural Information Processing Systems 23.

  • Nie, F., Wang, Z., Wang, R., & Li, X. (2019). Submanifold-preserving discriminant analysis with an auto-optimized graph. IEEE Transactions on Cybernetics, 50(8), 3682–3695.

    Article  Google Scholar 

  • Nie, F., Wang, Z., Wang, R., & Li, X. (2021). Adaptive local embedding learning for semi-supervised dimensionality reduction. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2021.3049371.

    Article  Google Scholar 

  • Nie, F., Wang, Z., Wang, R., Wang, Z., & Li, X. (2019). Towards robust discriminative projections learning via non-greedy \(\ell _{2,1}\)-norm minmax. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6), 2086–2100.

    Article  Google Scholar 

  • Oh, J. H., & Kwak, N. (2013). Generalization of linear discriminant analysis using lp-norm. Pattern Recognition Letters, 34(6), 679–685.

    Article  Google Scholar 

  • Pourahmadi, M. (2013). High-dimensional covariance estimation: With high-dimensional data (Vol. 882). Wiley.

    Book  Google Scholar 

  • Schäfer, J., & Strimmer, K. (2005). A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology, 4(1), 1–32.

    Article  Google Scholar 

  • Shewhart, W. A., & Wilks, S. S. (2006). Robust statistics: Theory and methods.

  • Stam, A. (1997). Nontraditional approaches to statistical classification: Some perspectives on l\(_p\)-norm methods. Annals of Operations Research, 74, 1–36.

    Article  Google Scholar 

  • Sun, Q., Xiang, S., & Ye, J. (2013). Robust principal component analysis via capped norms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 311–319).

  • Tebbens, J. D., & Schlesinger, P. (2007). Improving implementation of linear discriminant analysis for the high dimension/small sample size problem. Computational Statistics & Data Analysis, 52(1), 423–437.

    Article  Google Scholar 

  • Todorov, V. (2022). Scalable robust estimators with high breakdown point [r package rrcov version 1.7-0].

  • Todorov, V., & Filzmoser, P. (2010). An object-oriented framework for robust multivariate analysis. Journal of Statistical Software, 32, 1–47.

    Google Scholar 

  • Ünsal, M. G., & Nazman, E. (2020). Investigating socio-economic ranking of cities in turkey using data envelopment analysis (dea) and linear discriminant analysis (lda). Annals of Operations Research, 294(1), 281–295.

    Article  Google Scholar 

  • Wang, Z., Nie, F., Tian, L., Wang, R., & Li, X. (2020a). Discriminative feature selection via a structured sparse subspace learning module. In IJCAI (pp. 3009–3015).

  • Wang, H., Lu, X., Hu, Z., & Zheng, W. (2013). Fisher discriminant analysis with l1-norm. IEEE Transactions on Cybernetics, 44(6), 828–842.

    Article  Google Scholar 

  • Wang, Z., Nie, F., Zhang, C., Wang, R., & Li, X. (2020). Capped \(\ell _p\)-norm lda for outliers robust dimension reduction. IEEE Signal Processing Letters, 27, 1315–1319.

    Article  Google Scholar 

  • Ye, Q., Fu, L., Zhang, Z., Zhao, H., & Naiem, M. (2018). Lp-and ls-norm distance based robust linear discriminant analysis. Neural Networks, 105, 393–404.

    Article  Google Scholar 

  • Ye, Q., Yang, J., Liu, F., Zhao, C., Ye, N., & Yin, T. (2016). L1-norm distance linear discriminant analysis based on an effective iterative algorithm. IEEE Transactions on Circuits and Systems for Video Technology, 28(1), 114–129.

    Article  Google Scholar 

  • Zeiler, S., Nicheli, R., Ma, N., Brown, G. J., & Kolossa, D. (2016). Robust audiovisual speech recognition using noise-adaptive linear discriminant analysis. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 2797–2801). IEEE.

  • Zhang, D., Sun, Y., Ye, Q., & Tang, J. (2018). Recursive discriminative subspace learning with \(\ell _1\)-norm distance constraint. IEEE Transactions on Cybernetics, 50(5), 2138–2151.

    Article  Google Scholar 

  • Zhang, F., Yang, Z., Chen, Y., Yang, J., & Yang, G. (2018). Matrix completion via capped nuclear norm. IET Image Processing, 12(6), 959–966.

    Article  Google Scholar 

  • Zhao, M., Zhang, Z., Zhan, C., & Wang, W. (2017). Graph based semi-supervised classification via capped l2,1-norm regularized dictionary learning. In 2017 IEEE 15th international conference on industrial informatics (INDIN) (pp. 1019–1024). IEEE.

  • Zhao, J., Shi, L., & Zhu, J. (2014). Two-stage regularized linear discriminant analysis for 2-d data. IEEE Transactions on Neural Networks and Learning Systems, 26(8), 1669–1681.

    Article  Google Scholar 

  • Zheng, W., Lin, Z., & Wang, H. (2013). L1-norm kernel discriminant analysis via bayes error bound optimization for robust feature extraction. IEEE Transactions on Neural Networks and Learning Systems, 25(4), 793–805.

    Article  Google Scholar 

  • Zhong, F., & Zhang, J. (2013). Linear discriminant analysis based on l1-norm maximization. IEEE Transactions on Image Processing, 22(8), 3018–3027.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Management School, Hainan University, Haikou, 570228, People’s Republic of China

    Chun-Na Li & Yuan-Hai Shao

  2. School of Science, Hainan University, Haikou, 570228, People’s Republic of China

    Pei-Wei Ren

  3. College of Science, Zhejiang University of Science and Technology, Hangzhou, 310023, People’s Republic of China

    Yan-Ru Guo

  4. School of Economics, Zhejiang University of Technology, Hangzhou, 310023, People’s Republic of China

    Ya-Fen Ye

Authors
  1. Chun-Na Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pei-Wei Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan-Ru Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ya-Fen Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuan-Hai Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuan-Hai Shao.

Ethics declarations

Conflict of interest

The authors declare that there is not any conflict of interest in this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is supported by the Hainan Provincial Natural Science Foundation of China (No. 620QN234 and No. 120RC449), the National Natural Science Foundation of China (No. 62066012, No. 12101552, No. 12271131, No. 11871183 and No.61866010), and the Zhejiang Soft Science Research Project (2021C35003). .

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, CN., Ren, PW., Guo, YR. et al. Regularized linear discriminant analysis based on generalized capped \(l_{2,q}\)-norm. Ann Oper Res (2022). https://doi.org/10.1007/s10479-022-04959-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10479-022-04959-y

Keywords

Search

Navigation