Skip to main content
SpringerLink
Log in

Exact line and plane search for tensor optimization

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Line and plane searches are used as accelerators and globalization strategies in many optimization algorithms. We introduce a class of optimization problems called tensor optimization, which comprises applications ranging from tensor decompositions to least squares support tensor machines. We develop algorithms to efficiently compute the global minimizers of their line and plane search subproblems. Furthermore, we introduce scaled line and plane search, which compute an optimal scaling of the solution simultaneously with the optimal line or plane search step, and show that this scaling can be computed at almost no additional cost. Obtaining the global minimizers of (scaled) line and plane search problems often requires solving a bivariate or polyanalytic polynomial system. We show how to compute the isolated real solutions of bivariate polynomial systems and the isolated complex solutions of polyanalytic polynomial systems using a single generalized eigenvalue decomposition. Finally, we apply block term decompositions to the problem of blind multi-user detection-estimation in DS-CDMA communication to demonstrate that exact line and plane search can significantly reduce computation time of the workhorse tensor decomposition algorithm alternating least squares.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. The homogeneity requirement, which is satisfied by most tensor decompositions, is only necessary for scaled line and plane search.

References

  1. Abatzoglou, T.J.: Tensor-based techniques for the blind separation of DS-CDMA signals. Signal Process. 87(2), 322–336 (2007)

    Article  MathSciNet  Google Scholar 

  2. Acar, E., Rasmussen, M.A., Savorani, F., Næs, T., Bro, R.: Understanding data fusion within the framework of coupled matrix and tensor factorizations. Chemom. Intell. Lab. Syst. 129, 53–63 (2013)

    Article  Google Scholar 

  3. Berberich, E., Emeliyanenko, P., Sagraloff, M.: An elimination method for solving bivariate polynomial systems: Eliminating the usual drawbacks. In: Müller-Hannemann, M., Werneck, R. (eds.) Proceedings of the Thirteenth Workshop on Algorithm Engineering and Experiments (ALENEX), pp. 35–47. SIAM, San Francisco, CA, USA (2011)

    Chapter  Google Scholar 

  4. Bro, R.: Multi-way analysis in the food industry: models, algorithms, and applications. Ph.D. Thesis, University of Amsterdam (1998)

  5. Buchberger, B.: Gröbner bases and systems theory. Multidim. Syst. Signal Process. 12, 223–251 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  6. Busé, L., Khalil, H., Mourrain, B.: Resultant-based methods for plane curves intersection problems. In: Proceedings of the 8th International Conference on Computer Algebra in Scientific Computing. Lecture Notes in Computer Science, vol. 3718, pp. 75–92. Springer-Verlag, Heidelberg (2005)

  7. Byrd, R.H., Schnabel, R.B., Schultz, G.A.: Approximate solution of the trust region problem by minimization over two-dimensional subspaces. Math. Program. 40(1), 247–263 (1988)

    Article  MATH  Google Scholar 

  8. Carroll, J.D., Chang, J.J.: Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition. Psychometrika 35(3), 283–319 (1970)

    Article  MATH  Google Scholar 

  9. Chen, Y., Han, D., Qi, L.: New ALS methods with extrapolating search directions and optimal step size for complex-valued tensor decompositions. IEEE Trans. Signal Process. 59(12), 5888–5898 (2011)

    Article  MathSciNet  Google Scholar 

  10. Cichocki, A., Mandic, D., Caiafa, C., Phan, A.H., Zhou, G., Zhao, Q., De Lathauwer, L.: Tensor decompositions for signal processing applications. From two-way to multiway component analysis. IEEE Signal Process. Mag. 32(2), 145–163 (2015)

    Article  Google Scholar 

  11. Comon, P.: Independent component analysis, a new concept? Signal Process. 36(3), 287–314 (1994)

    Article  MATH  Google Scholar 

  12. Comon, P., Jutten, C.: Handbook of Blind Source Separation: Independent Component Analysis and Applications. Academic Press, New York (2010)

    Google Scholar 

  13. Corless, R.M., Gianni, P.M., Trager, B.M.: A reordered Schur factorization method for zero-dimensional polynomial systems with multiple roots. In: Proceedings of the International Symposium on Symbolic and Algebraic Computation, pp. 133–140. ACM Press, New York (1997)

  14. De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—Part I: lemmas for partitioned matrices. SIAM J. Matrix Anal. Appl. 30(3), 1022–1032 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  15. De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—Part II: definitions and uniqueness. SIAM J. Matrix Anal. Appl. 30(3), 1033–1066 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  16. De Lathauwer, L.: Blind separation of exponential polynomials and the decomposition of a tensor in rank-\(({L}_r,{L}_r,1)\) terms. SIAM J. Matrix Anal. Appl. 32(4), 1451–1474 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  17. De Lathauwer, L.: A short introduction to tensor-based methods for factor analysis and blind source separation. In: Proceeding of the 7th International Symposium on Image and Signal Processing and Analysis (ISPA 2011), pp. 558–563 (2011)

  18. De Lathauwer, L.: Block component analysis, a new concept for source separation and factor analysis. In: Latent Variable Analysis and Signal Separation. Lecture Notes in Computer Science, vol. 7191, pp. 1–8. Proceedings of 10th International Conference, LVA/ICA 2012, Tel Aviv, Israel, March 12–15, 2012. (2012)

  19. De Lathauwer, L., de Baynast, A.: Blind deconvolution of DS-CDMA signals by means of decomposition in rank-\((1, {L}, {L})\) terms. IEEE Trans. Signal Process. 56(4), 1562–1571 (2008)

    Article  MathSciNet  Google Scholar 

  20. De Lathauwer, L., Nion, D.: Decompositions of a higher-order tensor in block terms—Part III: alternating least squares algorithms. SIAM J. Matrix Anal. Appl. 30(3), 1067–1083 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  21. De Lathauwer, L., Vandewalle, J.: Dimensionality reduction in higher-order signal processing and rank-\(({R}_1,{R}_2,\ldots,{R}_{N})\) reduction in multilinear algebra. Linear Algebra Appl. 391, 31–55 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  22. Ding, Z., Johnson, C.R., Kennedy, R.A.: On the (non)existence of undesirable equilibria of godard blind equalizers. IEEE Trans. Signal Process. 40(10), 2425–2432 (1992)

    Article  MATH  Google Scholar 

  23. Diochnos, D.I., Emiris, I.Z., Tsigaridas, E.P.: On the asymptotic and practical complexity of solving bivariate systems over the reals. J. Symb. Comput. 44(7), 818–835 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  24. Domanov, I., De Lathauwer, L.: On the uniqueness of the canonical polyadic decomposition of third-order tensors—Part I: basic results and uniqueness of one factor matrix. SIAM J. Matrix Anal. Appl. 34, 855–875 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  25. Domanov, I., De Lathauwer, L.: On the uniqueness of the canonical polyadic decomposition of third-order tensors—Part II: overall uniqueness. SIAM J. Matrix Anal. Appl. 34, 876–903 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  26. Emiris, I.Z., Mourrain, B.: Matrices in elimination theory. J. Symb. Comput. 28(1–2), 3–43 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  27. Faugère, J.C.: A new efficient algorithm for computing Gröbner bases without reduction to zero (F5). In: Proceedings of the 2002 International Symposium on Symbolic and Algebraic Computation, ISSAC ’02, pp. 75–83. ACM, New York (2002)

  28. Godard, D.N.: Self-recovering equalization and carrier tracking in two-dimensional data communication systems. IEEE Trans. Commun. 28(11), 1867–1875 (1980)

    Article  Google Scholar 

  29. Gohberg, I.C., Lancaster, P., Rodman, L.: Matrix Polynomials. Academic Press, New York (1982)

    MATH  Google Scholar 

  30. Haardt, M., Roemer, F., Del Galdo, G.: Higher-order SVD-based subspace estimation to improve the parameter estimation accuracy in multidimensional harmonic retrieval problems. IEEE Trans. Signal Process. 56(7), 3198–3213 (2008)

    Article  MathSciNet  Google Scholar 

  31. Harshman, R.A.: Foundations of the PARAFAC procedure: models and conditions for an “explanatory” multi-modal factor analysis. UCLA Working Papers in Phonetics, vol. 16, pp. 1–84 (1970)

  32. Karami, A., Yazdi, M., Zolghadre Asli, A.: Noise reduction of hyperspectral images using kernel non-negative Tucker decomposition. IEEE J. Sel. Top. Signal Process. 5(3), 487–493 (2011)

  33. Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  34. Kruskal, J.B.: Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra Appl. 18(2), 95–138 (1977)

    Article  MATH  MathSciNet  Google Scholar 

  35. Kruskal, J.B.: Rank, Decomposition, and Uniqueness for 3-way and \({N}\)-way Arrays, pp. 7–18. Elsevier Science Publishers B.V., Amsterdam (1989)

  36. Kruskal, J.B., Harshman, R.A., Lundy, M.E.: How 3-MFA data can cause degenerate PARAFAC solutions, among other relationships. In: Coppi, R., Bolasco, S. (eds.) Multiway Data Analysis, pp. 115–121. North-Holland Publishing Co., Amsterdam (1989)

    Google Scholar 

  37. Li, T.Y.: Numerical solution of multivariate polynomial systems by homotopy continuation methods. Acta Numer. 6, 399–436 (1997)

    Article  Google Scholar 

  38. Lu, H., Plataniotis, K.N., Venetsanopoulos, A.N.: A survey of multilinear subspace learning for tensor data. Pattern Recognit. 44(7), 1540–1551 (2011)

    Article  MATH  Google Scholar 

  39. Manocha, D., Demmel, J.W.: Algorithms for intersecting parametric and algebraic curves II: multiple intersections. Graph. Models Image Process. 57(2), 81–100 (1995)

    Article  Google Scholar 

  40. Mitchell, B.C., Burdick, D.S.: Slowly converging PARAFAC sequences: swamps and two-factor degeneracies. J. Chem. 8(2), 155–168 (1994)

    Article  Google Scholar 

  41. Moore, R.E.: Methods and Applications of Interval Analysis. SIAM, Philadelphia (1987)

  42. Nion, D., De Lathauwer, L.: Block component model based blind DS-CDMA receivers. IEEE Trans. Signal Process. 56(11), 5567–5579 (2008)

    Article  MathSciNet  Google Scholar 

  43. Nion, D., De Lathauwer, L.: An enhanced line search scheme for complex-valued tensor decompositions. Appl. DS-CDMA Signal Process. 88(3), 749–755 (2008)

    Article  MATH  Google Scholar 

  44. Nion, D., De Lathauwer, L.: Tensor algebra and multidimensional harmonic retrieval in signal processing for MIMO radar. IEEE Trans. Signal Process. 58(11), 5693–5705 (2010)

    Article  MathSciNet  Google Scholar 

  45. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research, 2nd edn. Springer, New York (2006)

    MATH  Google Scholar 

  46. Rajih, M., Comon, P., Harshman, R.A.: Enhanced line search: a novel method to accelerate PARAFAC. SIAM J. Matrix Anal. Appl. 30(3), 1128–1147 (2008)

    Article  MathSciNet  Google Scholar 

  47. Šebek, M., Pejchová, S., Henrion, D., Kwakernaak, H.: Numerical methods for zeros and determinant of polynomial matrix. In: Proceedings of the IEEE Mediterranean Symposium on New Directions in Control and Automation, pp. 488–491. Chania, Crete, Greece (1996)

  48. Seidel, R., Wolpert, N.: On the exact computation of the topology of real algebraic curves. In: Proceedings of the Twenty-first Annual Symposium on Computational Geometry, SCG 2005, pp. 107–115. ACM, New York (2005)

  49. Sidiropoulos, N.D., Bro, R., Giannakis, G.B.: Parallel factor analysis in sensor array processing. IEEE Trans. Signal Process. 48(8), 2377–2388 (2000)

    Article  Google Scholar 

  50. Sidiropoulos, N.D., Dimić, G.Z.: Blind multiuser detection in W-CDMA systems with large delay spread. IEEE Signal Process. Lett. 8(3), 87–89 (2001)

    Article  Google Scholar 

  51. Sidiropoulos, N.D., Giannakis, G.B., Bro, R.: Blind PARAFAC receivers for DS-CDMA systems. IEEE Trans. Signal Process. 48(3), 810–823 (2000)

    Article  Google Scholar 

  52. Sorber, L., Van Barel, M., De Lathauwer, L.: Unconstrained optimization of real functions in complex variables. SIAM J. Optim. 22(3), 879–898 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  53. Sorber, L., Van Barel, M., De Lathauwer, L.: Numerical solution of bivariate and polyanalytic polynomial systems. SIAM J. Numer. Anal. 52(4), 1551–1572 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  54. Sorber, L., Van Barel, M., De Lathauwer, L.: Tensorlab v2.0. http://www.tensorlab.net/. Accessed January 2014

  55. Tao, D., Li, X., Wu, X., Hu, W., Maybank, J.: Supervised tensor learning. Knowl. Inform. Syst. 13(1), 1–42 (2007)

    Article  Google Scholar 

  56. Tucker, L.R.: Some mathematical notes on three-mode factor analysis. Psychometrika 31(3), 279–311 (1966)

    Article  MathSciNet  Google Scholar 

  57. Vasilescu, M.A.O., Terzopoulos, D.: Multilinear analysis of image ensembles: tensorfaces. In: Proceedings 7th European Conference on Computer Vision. Lecture Notes in Computer Science, vol. 2350, pp. 447–460. Springer-Verlag, Heidelberg (2002)

  58. Verschelde, J., Verlinden, J., Cools, R.: Homotopies exploiting newton polytopes for solving sparse polynomial systems. SIAM J. Numer. Anal. 31, 915–930 (1994)

    Article  MATH  MathSciNet  Google Scholar 

  59. Wilkinson, J.H.: The evaluation of the zeros of ill-conditioned polynomials. Part I. Numer. Math. 1(1), 150–166 (1959)

    Article  MATH  MathSciNet  Google Scholar 

  60. Zarzoso, V., Comon, P.: Optimal step-size constant modulus algorithm. IEEE Trans. Commun. 56(1), 10–13 (2008)

    Article  Google Scholar 

  61. Zhang, X., Ling, C., Qi, L.: The best rank-1 approximation of a symmetric tensor and related spherical optimization problems. SIAM J. Matrix Anal. Appl. 33(3), 806–821 (2012)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

L. Sorber was supported by a doctoral fellowship of the Flanders agency for Innovation by Science and Technology (IWT). I. Domanov and L. De Lathauwer were supported by the Research Council KU Leuven: GOA/10/09 MaNet, CoE PFV/10/002 (OPTEC), PDM postdoc Grant; F.W.O.: project G.0830.14N, G.0881.14N; the Belgian Federal Science Policy Office: IUAP P7 (DYSCO II, Dynamical systems, control and optimization, 2012-2017); EU: The research leading to these results has received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013) / ERC Advanced Grant: BIOTENSORS (no. 339804). This paper reflects only the authors’ views and the Union is not liable for any use that may be made of the contained information. M. Van Barel was supported by the Research Council KU Leuven: OT/10/038, CoE PF/10/002 (OPTEC); F.W.O.: project G.0828.14N; by the Belgian Network DYSCO (Dynamical Systems, Control, and Optimization), funded by the Interuniversity Attraction Poles Programme, initiated by the Belgian State, Science Policy Office.

Author information

Authors and Affiliations

  1. Department of Computer Science, KU Leuven, Celestijnenlaan 200A, 3001, Leuven, Belgium

    Laurent Sorber & Marc Van Barel

  2. Group Science, Engineering and Technology, KU Leuven–Kulak, E. Sabbelaan 53, 8500, Kortrijk, Belgium

    Ignat Domanov & Lieven De Lathauwer

  3. Department of Electrical Engineering, ESAT/STADIUS KU Leuven, Kasteelpark Arenberg 10, bus 2446, 3001, Leuven-Heverlee, Belgium

    Laurent Sorber, Ignat Domanov & Lieven De Lathauwer

  4. iMinds Medical IT, Leuven, Belgium

    Laurent Sorber, Ignat Domanov & Lieven De Lathauwer

Authors
  1. Laurent Sorber
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ignat Domanov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marc Van Barel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lieven De Lathauwer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ignat Domanov.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sorber, L., Domanov, I., Van Barel, M. et al. Exact line and plane search for tensor optimization. Comput Optim Appl 63, 121–142 (2016). https://doi.org/10.1007/s10589-015-9761-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-015-9761-5

Keywords

Search

Navigation