Skip to main content
SpringerLink
Log in

Iterative Methods Based on Soft Thresholding of Hierarchical Tensors

  • Published:
Foundations of Computational Mathematics Aims and scope Submit manuscript

Abstract

We construct a soft thresholding operation for rank reduction in hierarchical tensors and subsequently consider its use in iterative thresholding methods, in particular for the solution of discretized high-dimensional elliptic problems. The proposed method for the latter case adjusts the thresholding parameters, by an a posteriori criterion requiring only bounds on the spectrum of the operator, such that the arising tensor ranks of the resulting iterates remain quasi-optimal with respect to the algebraic or exponential-type decay of the hierarchical singular values of the true solution. In addition, we give a modified algorithm using inexactly evaluated residuals that retains these features. The effectiveness of the scheme is demonstrated in numerical experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Alternative notions of nuclear norms of tensors can be defined by duality [53], but these are more difficult to handle.

  2. All methods were implemented in C++, using LAPACK for numerical linear algebra operations.

References

  1. Bachmayr, M., Cohen, A.: Kolmogorov widths and low-rank approximations of parametric elliptic PDEs. Math. Comp. (2016). In press.

  2. Bachmayr, M., Dahmen, W.: Adaptive near-optimal rank tensor approximation for high-dimensional operator equations. Found. Comput. Math. 15(4), 839–898 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bachmayr, M., Dahmen, W.: Adaptive low-rank methods: Problems on Sobolev spaces. SIAM J. Numer. Anal. (2016). In press.

  4. Ballani, J., Grasedyck, L.: A projection method to solve linear systems in tensor format. Numer. Linear Algebra Appl. 20(1), 27–43 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  5. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sciences 2, 182–202 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Beylkin, G., Mohlenkamp, M. J.: Algorithms for numerical analysis in high dimensions. SIAM J. Sci. Comput. 26(6), 2133–2159 (electronic) (2005)

  7. Billaud-Friess, M., Nouy, A., Zahm, O.: A tensor approximation method based on ideal minimal residual formulations for the solution of high-dimensional problems. ESAIM: M2AN 48(6), 1777–1806 (2014).

  8. Bredies, K., Lorenz, D. A.: Linear convergence of iterative soft-thresholding. J Fourier Anal Appl 14, 813–837 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  9. Cai, J.-F., Candès, E. J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J Optimization 20, 1956–1982 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  10. Cancès, E., Ehrlacher, V., Lelièvre, T.: Convergence of a greedy algorithm for high-dimensional convex nonlinear problems. Math. Models Methods Appl. Sci. 21(12), 2433–2467 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  11. Cohen, A., Dahmen, W., DeVore, R.: Adaptive wavelet methods for elliptic operator equations: Convergence rates. Mathematics of Computation 70(233), 27–75 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  12. Cohen, A., DeVore, R., Schwab, C.: Analytic regularity and polynomial approximation of parametric and stochastic PDE’s. Analysis and Applications 9, 1–37 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  13. Da Silva, C., Herrmann, F. J.: Optimization on the hierarchical Tucker manifold—applications to tensor completion. Linear Algebra Appl. 481, 131–173 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  14. Dahlke, S., Fornasier, M., Raasch, T.: Multilevel preconditioning and adaptive sparse solution of inverse problems. Math. Comp. 81, 419–446 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  15. Dahmen, W., DeVore, R., Grasedyck, L., Süli, E.: Tensor-sparsity of solutions to high-dimensional elliptic partial differential equations. Found. Comput. Math. (2015). In press. DOI:10.1007/s10208-015-9265-9

  16. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. Pure Appl. Math. 57(11), 1413–1457 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  17. DeVore, R.: Nonlinear approximation. Acta Numer. 7, 51–150 (1998)

    Article  MATH  Google Scholar 

  18. Dolgov, S. V., Savostyanov, D. V.: Alternating minimal energy methods for linear systems in higher dimensions. SIAM J. Sci. Comput. 36(5), A2248–A2271 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  19. Donovan, G. C., Geronimo, J. S., Hardin, D. P.: Orthogonal polynomials and the construction of piecewise polynomial smooth wavelets. SIAM J. Math. Anal. 30, 1029–1056 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  20. Falcó, A., Nouy, A.: Proper generalized decomposition for nonlinear convex problems in tensor Banach spaces. Numer. Math. 121(3), 503–530 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  21. Gandy, S., Recht, B., Yamada, I.: Tensor completion and low-n-rank tensor recovery via convex optimization. Inverse Problems 27(2), 025010 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  22. Gantumur, T., Harbrecht, H., Stevenson, R.: An optimal adaptive wavelet method without coarsening of the iterands. Math. Comp. 76(258), 615–629 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  23. Grasedyck, L.: Existence and computation of low Kronecker-rank approximations for large linear systems of tensor product structure. Computing 72, 247–265 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  24. Grasedyck, L.: Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl. 31(4), 2029–2054 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  25. Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitt. 36(1), 53–78 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  26. Hackbusch, W.: Entwicklungen nach Exponentialsummen. Tech. Rep. 4, MPI Leipzig (2005)

  27. Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics, vol. 42. Springer-Verlag Berlin Heidelberg (2012)

    Book  MATH  Google Scholar 

  28. Hackbusch, W.: Numerical tensor calculus. Acta Numer. 23, 651–742 (2014)

    Article  MathSciNet  Google Scholar 

  29. Hackbusch, W., Khoromskij, B., Tyrtyshnikov, E.: Approximate iterations for structured matrices. Numerische Mathematik 109, 119–156 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  30. Hackbusch, W., Kühn, S.: A new scheme for the tensor representation. J. Fourier Anal. Appl. 15(5), 706–722 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  31. Holtz, S., Rohwedder, T., Schneider, R.: The alternating linear scheme for tensor optimization in the tensor train format. SIAM J. Sci. Comput. 34(2), A683–A713 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  32. Huang, B., Mu, C., Goldfarb, D., Wright, J.: Provable models for robust low-rank tensor completion. Pacific Journal of Optimization 11(2), 339–364 (2015)

    MathSciNet  MATH  Google Scholar 

  33. Khoromskij, B. N.: Tensor-structured preconditioners and approximate inverse of elliptic operators in \({\mathbb{R}^{d}}\). Constr. Approx. 30(3), 599–620 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  34. Khoromskij, B. N.: Tensor numerical methods for multidimensional: PDEs theoretical analysis and initial applications. ESAIM: ProcS 48, 1–28 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  35. Khoromskij, B. N., Schwab, C.: Tensor-structured Galerkin approximation of parametric and stochastic elliptic PDEs. SIAM J. Sci. Comput. 33(1), 364–385 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  36. Kressner, D., Steinlechner, M., Vandereycken, B.: Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure. preprint (2015)

  37. Kressner, D., Tobler, C.: Low-rank tensor Krylov subspace methods for parametrized linear systems. SIAM J. Matrix Anal. Appl. 32, 1288–1316 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  38. Kressner, D., Uschmajew, A.: On low-rank approximability of solutions to high-dimensional operator equations and eigenvalue problems. Linear Algebra Appl. 493, 556–572 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  39. Kühn, S.: Hierarchische Tensordarstellung. Ph.D. thesis, Universität Leipzig (2012)

  40. Lai, M.-J., Yin, W.: Augmented \(\ell _1\) and nuclear-norm models with a globally linearly convergent algorithm. SIAM Journal on Imaging Sciences 6(2), 1059–1091 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  41. Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 208–220 (2013)

    Article  Google Scholar 

  42. Ma, S., Goldfarb, D., Chen, L.: Fixed point and Bregman iterative methods for matrix rank minimization. Mathematical Programming 128(1-2), 321–353 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  43. Markus, A. S.: The eigen- and singular values of the sum and product of linear operators. Russian Mathematical Surveys 19(4), 91–120 (1964)

    Article  MATH  Google Scholar 

  44. Mirsky, L.: Symmetric gauge functions and unitarily invariant norms. Quarterly Journal of Mathematics 11, 50–59 (1960)

    Article  MathSciNet  MATH  Google Scholar 

  45. Moreau, J.: Proximité et dualité dans un espace hilbertien. Bulletin de la Société Mathématique de France 93, 273–299 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  46. Oseledets, I., Tyrtyshnikov, E.: Breaking the curse of dimensionality, or how to use SVD in many dimensions. SIAM J. Sci. Comput. 31(5), 3744–3759 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  47. Oseledets, I. V.: Tensor-train decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  48. Recht, B., Fazel, M., Parrilo, P. A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review 52(3), 471–501 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  49. Schneider, R., Uschmajew, A.: Approximation rates for the hierarchical tensor format in periodic Sobolev spaces. J. Complexity 30(2), 56–71 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  50. Vidal, G.: Efficient classical simulation of slightly entangled quantum computations. Phys. Rev. Lett. 91, 147902 (2003)

    Article  Google Scholar 

  51. White, S. R.: Density matrix formulation for quantum renormalization groups. Phys. Rev. Lett. 69, 2863–2866 (1992)

    Article  Google Scholar 

  52. Wright, S. J., Nowak, R. D., Figueiredo, M. A. T.: Sparse reconstruction by separable approximation. IEEE Trans Sig. Process. 57, 2479–2493 (2009)

    Article  MathSciNet  Google Scholar 

  53. Yuan, M., Zhang, C.-H.: On tensor completion via nuclear norm minimization. Found. Comput. Math. (2015). In press. DOI:10.1007/s10208-015-9269-5

Download references

Acknowledgments

This research was supported in part by DFG SPP 1324 and ERC AdG BREAD.

Author information

Authors and Affiliations

  1. Sorbonne Universités, UPMC Univ Paris 06, CNRS, UMR 7598, Laboratoire Jacques-Louis Lions, 4 place Jussieu, 75005, Paris, France

    Markus Bachmayr

  2. Institut für Mathematik, Technische Universität Berlin, Straße des 17. Juni 136, 10623, Berlin, Germany

    Reinhold Schneider

Authors
  1. Markus Bachmayr
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reinhold Schneider
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Markus Bachmayr.

Additional information

Communicated by Endre Süli.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bachmayr, M., Schneider, R. Iterative Methods Based on Soft Thresholding of Hierarchical Tensors. Found Comput Math 17, 1037–1083 (2017). https://doi.org/10.1007/s10208-016-9314-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10208-016-9314-z

Keywords

Mathematics Subject Classification

Search

Navigation