Abstract
In this paper we derive a family of new extended SMART (Simultaneous Multiplicative Algebraic Reconstruction Technique) algorithms for Non-negative Matrix Factorization (NMF). The proposed algorithms are characterized by improved efficiency and convergence rate and can be applied for various distributions of data and additive noise. Information theory and information geometry play key roles in the derivation of new algorithms. We discuss several loss functions used in information theory which allow us to obtain generalized forms of multiplicative NMF learning adaptive algorithms. We also provide flexible and relaxed forms of the NMF algorithms to increase convergence speed and impose an additional constraint of sparsity. The scope of these results is vast since discussed generalized divergence functions include a large number of useful loss functions such as the Amari α– divergence, Relative entropy, Bose-Einstein divergence, Jensen-Shannon divergence, J-divergence, Arithmetic-Geometric (AG) Taneja divergence, etc. We applied the developed algorithms successfully to Blind (or semi blind) Source Separation (BSS) where sources may be generally statistically dependent, however are subject to additional constraints such as nonnegativity and sparsity. Moreover, we applied a novel multilayer NMF strategy which improves performance of the most proposed algorithms.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Lee, D.D., Seung, H.S.: Learning of the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999)
Cho, Y.-C., Choi, S.: Nonnegative features of spectro-temporal sounds for classification. Pattern Recognition Letters 26(9), 1327–1336 (2005)
Sajda, P., Du, S., Parra, L.: Recovery of constituent spectra using non-negative matrix factorization. In: Proceedings of SPIE. Wavelets: Applications in Signal and Image Processing, vol. 5207, pp. 321–331 (2003)
Guillamet, D., Vitri‘a, J., Schiele, B.: Introducing a weighted nonnegative matrix factorization for image classification. Pattern Recognition Letters 24, 2447 (2004)
Li, D., Adali, T., Wang, D.E.W.: Non-negative matrix factorization with orthogonality constraints for chemical agent detection in Raman spectra. In: IEEE Workshop on Machine Learning for Signal Processing, Mystic USA (2005)
Cichocki, A., Zdunek, R., Amari, S.: Csiszar’s divergences for non-negative matrix factorization: Family of new algorithms. In: Rosca, J.P., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA 2006. LNCS, vol. 3889, pp. 32–39. Springer, Heidelberg (2006)
Paatero, P., Tapper, U.: Positive matrix factorization: A nonnegative factor model with optimal utilization of error estimates of data values. Environmetrics 5, 111–126 (1994)
Oja, E., Plumbley, M.: Blind separation of positive sources using nonnegative PCA. In: 4th International Symposium on Independent Component Analysis and Blind Signal Separation, Nara, Japan (2003)
Hoyer, P.: Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research 5, 1457–1469 (2004)
Kompass, R.: A generalized divergence measure for nonnegative matrix factorization, Neuroinfomatics Workshop, Torun, Poland (2005)
Dhillon, I., Sra, S.: Generalized nonnegative matrix approximations with Bregman divergences. In: NIPS -Neural Information Proc. Systems, Vancouver Canada (2005)
Berry, M., Browne, M., Langville, A., Pauca, P., Plemmons, R.: Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics and Data Analysis (2006), http://www.wfu.edu/~plemmons/papers.htm
Lee, D.D., Seung, H.S.: Algorithms for nonnegative matrix factorization, vol. 13, pp. 556–562. NIPS, MIT Press, Cambridge (2001)
Novak, M., Mammone, R.: Use of nonnegative matrix factorization for language model adaptation in a lecture transcription task. In: Proceedings of the 2001 IEEE Conference on Acoustics, Speech and Signal Processing, Salt Lake City, UT, vol. 1, pp. 541–544 (2001)
Feng, T., Li, S.Z., Shum, H.Y., Zhang, H.: Local nonnegative matrix factorization as a visual representation. In: Proceedings of the 2nd International Conference on Development and Learning, Cambridge, MA, pp. 178–193 (2002)
Chen, Z., Cichocki, A., Rutkowski, T.: Constrained non-negative matrix factorization method for EEG analysis in early detection of Alzheimer’s disease. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2006, Toulouse, France (2006)
Cichocki, A., Amari, S.: Adaptive Blind Signal And Image Processing (New revised and improved edition). John Wiley, New York (2003)
Cichocki, A., Zdunek, R.: NMFLAB Toolboxes for Signal and Image Processing, Japan (2006), www.bsp.brain.riken.go.jp
Merritt, M., Zhang, Y.: An interior-point gradient method for large-scale totally nonnegative least squares problems. Technical report, Department of Computational and Applied Mathematics, Rice University, Houston, Texas, USA (2004)
Minami, M., Eguchi, S.: Robust blind source separation by beta-divergence. Neural Computation 14, 1859–1886 (2002)
Jorgensen, B.: The Theory of Dispersion Models. Chapman and Hall, Boca Raton (1997)
Csiszár, I.: Information measures: A critical survey. Prague Conference on Information Theory, Academia Prague A, 73–86 (1974)
Amari, S., Nagaoka, H.: Methods of Information Geometry. Oxford University Press, Oxford (2000)
Zhang, J.: Divergence function, duality and convex analysis. Neural Computation 16, 159–195 (2004)
Schraudolf, N.: Gradient-based manipulation of non-parametric entropy estimates. IEEE Trans. on Neural Networks 16, 159–195 (2004)
Byrne, C.: Accelerating the EMML algorithm and related iterative algorithms by rescaled block-iterative (RBI) methods. IEEE Transactions on Image Processing 7, 100–109 (1998)
Byrne, C.: Choosing parameters in block-iterative or ordered subset reconstruction algorithms. IEEE Transactions on Image Progressing 14, 321–327 (2005)
Amari, S.: Differential-Geometrical Methods in Statistics. Springer, Heidelberg (1985)
Amari, S.: Information geometry of the EM and em algorithms for neural networks. neural networks 8(9), 1379–1408 (1995)
Cressie, N.A., Read, T.C.R.: Goodness-of-Fit Statistics for Discrete Multivariate Data. Springer, Heidelberg (1988)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cichocki, A., Amari, Si., Zdunek, R., Kompass, R., Hori, G., He, Z. (2006). Extended SMART Algorithms for Non-negative Matrix Factorization. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds) Artificial Intelligence and Soft Computing – ICAISC 2006. ICAISC 2006. Lecture Notes in Computer Science(), vol 4029. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11785231_58
Download citation
DOI: https://doi.org/10.1007/11785231_58
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-35748-3
Online ISBN: 978-3-540-35750-6
eBook Packages: Computer ScienceComputer Science (R0)