Skip to main content
SpringerLink
Log in

Parameterizations make different model selections: Empirical findings from factor analysis

  • Research Article
  • Published:
Frontiers of Electrical and Electronic Engineering in China

Abstract

How parameterizations affect model selection performance is an issue that has been ignored or seldom studied since traditional model selection criteria, such as Akaike’s information criterion (AIC), Schwarz’s Bayesian information criterion (BIC), difference of negative log-likelihood (DNLL), etc., perform equivalently on different parameterizations that have equivalent likelihood functions. For factor analysis (FA), in addition to one traditional model (shortly denoted by FA-a), it was previously found that there is another parameterization (shortly denoted by FA-b) and the Bayesian Ying-Yang (BYY) harmony learning gets different model selection performances on FA-a and FA-b. This paper investigates a family of FA parameterizations that have equivalent likelihood functions, where each one (shortly denoted by FA-r) is featured by an integer r, with FA-a as one end that r = 0 and FA-b as the other end that r reaches its upper-bound. In addition to the BYY learning in comparison with AIC, BIC, and DNLL, we also implement variational Bayes (VB). Several empirical finds have been obtained via extensive experiments. First, both BYY and VB perform obviously better on FA-b than on FA-a, and this superiority of FA-b is reliable and robust. Second, both BYY and VB outperform AIC, BIC, and DNLL, while BYY further outperforms VB considerably, especially on FA-b. Moreover, with FA-a replaced by FA-b, the gain obtained by BYY is obviously higher than the one by VB, while the gain by VB is better than no gain by AIC, BIC, and DNLL. Third, this paper also demonstrates how each part of priors incrementally and jointly improves the performances, and further shows that using VB to optimize the hyperparameters of priors deteriorates the performances while using BYY for this purpose can further improve the performances.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control, 1974, 19(6): 716–723

    Article  MathSciNet  MATH  Google Scholar 

  2. Schwarz G. Estimating the dimension of a model. Annals of Statistics, 1978, 6(2): 461–464

    Article  MathSciNet  MATH  Google Scholar 

  3. Rissanen J. Modelling by the shortest data description. Automatica, 1978, 14(5): 465–471

    Article  MATH  Google Scholar 

  4. Anderson T, Rubin H. Statistical inference in factor analysis. In: Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability. 1956, 5: 111–150

    MathSciNet  Google Scholar 

  5. Fodor I K. A survey of dimension reduction techniques. Technical Report UCRL-ID-148494. 2002

  6. Jolliffe I T. Principal Component Analysis. 2nd ed. New York: Springer, 2002

    MATH  Google Scholar 

  7. Tipping M E, Bishop C M. Mixtures of probabilistic principal component analyzers. Neural Computation, 1999, 11(2): 443–482

    Article  Google Scholar 

  8. Bishop C M. Variational principal components. In: Proceedings of the Ninth International Conference on Artificial Neural Networks. 1999, 509–514

  9. Ghahramani Z, Beal M J. Variational inference for Bayesian mixtures of factor analysers. Advances in Neural Information Processing System, 2000, 12: 449–455

    Google Scholar 

  10. Nielsen F B. Variational approach to factor analysis and related models. Dissertation for the Master’s Degree. Lyngby: Technical University of Denmark, 2004

    Google Scholar 

  11. Hills S E, Smith A F. Parameterization issues in Bayesian inference. Bayesian Statistics, 1992, 4: 227–246

    MathSciNet  Google Scholar 

  12. Kass R E, Slate E H. Reparameterization and diagnostics of posterior nonnormality. Bayesian Statistics, 1992, 4: 289–305

    MathSciNet  Google Scholar 

  13. Gelman A. Parameterization and Bayesian modeling. Journal of the American Statistical Association, 2004, 99(466): 537–545

    Article  MathSciNet  MATH  Google Scholar 

  14. Ghosh J, Dunson D B. Default prior distributions and efficient posterior computation in Bayesian factor analysis. Journal of Computational and Graphical Statistical Statistics, 2009, 18(2): 306–320

    Article  Google Scholar 

  15. Xu L. Bayesian Ying Yang system and theory as a unified statistical learning approach: (i) unsupervised and semiunsupervised learning. In: Amari S, Kassabov N, eds. Brainlike Computing and Intelligent Information Systems. Berlin: Springer-Verlag, 1997, 241–274

    Google Scholar 

  16. Xu L. Bayesian Ying-Yang learning theory for data dimension reduction and determination. Journal of Computational Intelligence in Finance, 1998, 6(5): 6–18

    Google Scholar 

  17. Xu L. BYY harmony learning, independent state space, and generalized APT financial analyses. IEEE Transactions on Neural Networks, 2001, 12(4): 822–849

    Article  Google Scholar 

  18. Hu X, Xu L. A comparative investigation on subspace dimension determination. Neural Networks, 2004, 17(8–9): 1051–1059

    Article  MathSciNet  MATH  Google Scholar 

  19. Shi L, Xu L. Local factor analysis with automatic model selection: a comparative study and digits recognition application. In: Proceedings Part II of the 16th International Conference on Artificial Neural Networks. 2006, 260–269

  20. Jordan M I, Ghahramani Z, Jaakkola T S, Saul L K. An introduction to variational methods for graphical models. Machine Learning, 1999, 37(2): 183–233

    Article  MATH  Google Scholar 

  21. Beal M J. Variational algorithms for approximate Bayesian inference. Dissertation for the Doctoral Degree. London: University College London, 2003

    Google Scholar 

  22. Xu L. Bayesian Ying-Yang system, best harmony learning, and five action circling. Frontiers of Electrical and Electronic Engineering in China, 2010, 5(3): 281–328

    Article  Google Scholar 

  23. Xu L. Fundamentals, challenges, and advances of statistical learning for knowledge discovery and problem solving: a BYY harmony perspective. In: Proceedings of International Conference on Neural Networks and Brain. 2005, 1: 24–55

    Google Scholar 

  24. Rubin D B, Thayer D T. EM algorithm for ML factor analysis. Psychometrika, 1982, 47(1): 69–76

    Article  MathSciNet  MATH  Google Scholar 

  25. Bozdogan H, Ramirez D E. FACAIC: model selection algorithm for the orthogonal factor model using AIC and FACAIC. Psychometrika, 1988, 53(3): 407–415

    Article  MATH  Google Scholar 

  26. Tu S, Xu L. A study of several model selection criteria for determining the number of signals. In: Proceedings of IEEE International Conference on Acoustics Speech and Signal Processing. 2010, 1966–1969

  27. Xu L. Bayesian-Kullback coupled YING-YANG machines: unified learning and new results on vector quantization. In: Proceedings of International Conference on Neural Information Processing. 1995, 977–988

  28. Xu L. Codimensional matrix pairing perspective of BYY harmony learning: hierarchy of bilinear systems, joint decomposition of data-covariance, and applications of network biology. Frontiers of Electrical and Electronic Engineering in China, 2011, 6(1): 86–119

    Article  Google Scholar 

  29. Xu L. Bayesian Ying Yang learning. Scholarpedia, 2007, 2(3): 1809

    Article  Google Scholar 

  30. Xu L. Bayesian Ying Yang system, best harmony learning, and Gaussian manifold based family. In: Zurada J, Yen G, Wang J, eds. Computational Intelligence: Research Frontiers. Berlin-Heidelberg: Springer-Verlag, 2008, 5050: 48–78

    Chapter  Google Scholar 

  31. Tu S, Xu L. An investigation of several typical model selection criteria for detecting the number of signals. Frontiers of Electrical and Electronic Engineering in China, 2011 (in Press)

  32. Asuncion A, Newman D. UCI machine learning repository, 2007. http://www.ics.uci.edu/~mlearn/MLRepository.html

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China

    Shikui Tu & Lei Xu

Authors
  1. Shikui Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Xu.

Additional information

Shikui TU is a Ph.D candidate of the Department of Computer Science and Engineering, The Chinese University of Hong Kong. He obtained his Bachelor degree from School of Mathematical Science, Peking University, in 2006. His research interests include statistical learning, pattern recognition, and bioinformatics.

Lei XU, chair professor of The Chinese University of Hong Kong (CUHK), Fellow of IEEE (2001–), Fellow of International Association for Pattern Recognition (2002–), and Academician of European Academy of Sciences (2002–). He completed his Ph.D thesis at Tsinghua University by the end of 1986, became postdoc at Peking University in 1987, then promoted to associate professor in 1988 and a professor in 1992. During 1989–1993 he was research associate and postdoc in Finland, Canada and USA, including Harvard and MIT. He joined CUHK as senior lecturer in 1993, professor in 1996, and chair professor in 2002. He published several well-cited papers on neural networks, statistical learning, and pattern recognition, e.g., his papers got over 3400 citations (SCI) and over 6300 citations by Google Scholar (GS), with the top-10 papers scored over 2100 (SCI) and 4100 (GS). One paper scored 790 (SCI) and 1351 (GS). He served as a past governor of International Neural Network Society (INNS), a past president of APNNA, and a member of Fellow Committee of IEEE CI Society. He received several national and international academic awards (e.g., 1993 National Nature Science Award, 1995 INNS Leadership Award and 2006 APNNA Outstanding Achievement Award).

About this article

Cite this article

Tu, S., Xu, L. Parameterizations make different model selections: Empirical findings from factor analysis. Front. Electr. Electron. Eng. China 6, 256–274 (2011). https://doi.org/10.1007/s11460-011-0150-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11460-011-0150-2

Keywords

Search

Navigation