Skip to main content
Log in

A partially parallel splitting method for multiple-block separable convex programming with applications to robust PCA

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

We consider a multiple-block separable convex programming problem, where the objective function is the sum of m individual convex functions without overlapping variables, and the constraints are linear, aside from side constraints. Based on the combination of the classical Gauss–Seidel and the Jacobian decompositions of the augmented Lagrangian function, we propose a partially parallel splitting method, which differs from existing augmented Lagrangian based splitting methods in the sense that such an approach simplifies the iterative scheme significantly by removing the potentially expensive correction step. Furthermore, a relaxation step, whose computational cost is negligible, can be incorporated into the proposed method to improve its practical performance. Theoretically, we establish global convergence of the new method in the framework of proximal point algorithm and worst-case nonasymptotic \({\mathcal {O}}(1/t)\) convergence rate results in both ergodic and nonergodic senses, where t counts the iteration. The efficiency of the proposed method is further demonstrated through numerical results on robust PCA, i.e., factorizing from incomplete information of an unknown matrix into its low-rank and sparse components, with both synthetic and real data of extracting the background from a corrupted surveillance video.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Bardsley, J., Knepper, S., Nagy, J.: Structured linear algebra problems in adaptive optics imaging. Adv. Comput. Math. 35, 103–117 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  2. Bose, N., Boo, K.: High-resolution image reconstruction with multisensors. Int. J. Imaging Syst. Technol. 9, 294–304 (1998)

    Article  Google Scholar 

  3. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010)

    Article  MATH  Google Scholar 

  4. Cai, J., Candés, E., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20, 1956–1982 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  5. Cai, J., Chan, R., Nikolova, M.: Two-phase approach for deblurring images corrupted by impulse plus gaussian noise. Inverse Probl. Imaging 2, 187–204 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  6. Cai, X.J., Gu, G.Y., He, B.S., Yuan, X.M.: A proximal point algorithm revisit on the alternating direction method of multipliers. Sci. China Math. 56, 2179–2186 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  7. Candés, E., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58, 1–37 (2011)

    Article  Google Scholar 

  8. Chan, R.H., Yang, J.F., Yuan, X.M.: Alternating direction method for image inpainting in wavelet domain. SIAM J. Imaging Sci. 4(3), 807–826 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  9. Chandrasekaran, V., Sanghavi, S., Parrilo, P., Willskyc, A.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21, 572–596 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  10. Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Prog., Ser. A (2014). doi:10.1007/s10107-014-0826-5

  11. Dinh, D., Savorgnan, C., Diehl, M.: Combining Lagrangian decomposition and excessive gap smoothing technique for solving large-scale separable convex optimization problems. Comput. Optim. Appl. 55, 75–111 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  12. Dinh, Q., Necoara, I., Savorgnan, C., Diehl, M.: An inexact perturbed path-following method for Lagrangian decomposition in large-scale separable convex optimization. SIAM J. Optim. 23, 95–125 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  13. Eckstein, J.: Splitting methods for monotone operators with applications to parallel optimization. Ph.D. thesis, Massachusetts Institute of Technology (1989)

  14. Eckstein, J.: Augmented Lagrangian and alternating direction methods for convex optimization: A tutorial and some illustrative computational results. Tech. Rep. 32-2012, Rutgers University (2012)

  15. Eckstein, J., Bertsekas, D.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    Article  MATH  MathSciNet  Google Scholar 

  16. Facchinei, F., Pang, J.: Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, New York (2003)

    Google Scholar 

  17. Fazel, M., Goodman, J.: Approximations for partially coherent optical imaging systems. Stanford University, Tech. rep. (1998)

    Google Scholar 

  18. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 16–40 (1976)

    Article  Google Scholar 

  19. Glowinski, R.: On alternating direction methods of multipliers: a historical perspective. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanmäki, P. (eds.) Modeling, Simulation and Optimization for Science and Technology, Computational Methods in Applied Sciences, chap. 4, vol. 4, pp. 59–82. Springer, New York (2014)

    Google Scholar 

  20. Glowinski, R., Marrocco, A.: Approximation par éléments finis d’ordre un et résolution par pénalisation-dualité d’une classe de problèmes non linéaires. R.A.I.R.O R2, 41–76 (1975)

    MathSciNet  Google Scholar 

  21. Goldstein, T., Osher, S.: The split Bregman method for \(\ell _1\) regularized problems. SIAM J. Imaging Sci. 2, 323–343 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  22. Gu, G., He, B., Yuan, X.: Customized proximal point algorithms for linearly constrained convex minimization and saddle-point problems: a unified approach. Comput. Optim. Appl. 59, 135–161 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  23. Han, D., He, H., Yang, H., Yuan, X.: A customized Douglas–Rachford splitting algorithm for separable convex minimization with linear constraints. Numer. Math. 127, 167–200 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  24. Han, D., Kong, W., Zhang, W.: A partial splitting augmented Lagrangian method for low patch-rank image decomposition. J. Math. Imaging. Vis. 51, 145–160 (2015)

    Article  MathSciNet  Google Scholar 

  25. Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim. Theory. Appl. 55, 227–238 (2012)

    Article  MathSciNet  Google Scholar 

  26. Han, D., Yuan, X., Zhang, W.: An augmented-Lagrangian-based parallel splitting method for separable convex minimization with applications to image processing. Math. Comput. 83, 2263–2291 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  27. He, B.: Parallel splitting augmented Lagrangian methods for monotone structured variational inequalities. Comput. Optim. Appl. 42, 195–212 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  28. He, B., Hou, L., Yuan, X.: On full Jacobian decomposition of the augmented Lagrangian method for separable convex programming (2013). Optimization Online

  29. He, B., Tao, M., Xu, M., Yuan, X.: Alternating directions based contraction method for generally separable linearly constrained convex programming problems. Optimizaition 62, 573–596 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  30. He, B., Tao, M., Yuan, X.: A splitting method for separate convex programming with linking linear constraints. IMA J. Numer. Anal. 31, 394–426 (2015)

    Article  MathSciNet  Google Scholar 

  31. He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian-back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  32. He, B., Yuan, X.: On the direct extension of ADMM for multi-block separable convex programming and beyond: from variational inequality perspective. Optimization Online. http://www.optimizationonline.org/DB_FILE/2014/03/4293.pdf (2014)

  33. He, B., Yuan, X.: On the \(O(1/n)\) convergence rate of Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  34. Hestenes, M.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    Article  MATH  MathSciNet  Google Scholar 

  35. Hong, M., Luo, Z.: On the linear convergence of the alternating direction method of multipliers. arXiv:1208.3922v3

  36. Larsen, R.: PROPACK-software for large and sparse SVD calculations. http://sun.stanford.edu/srmunk/PROPACK/

  37. Lauritzen, S.L.: Graphical Models. Oxford University Press, New York (1996)

    Google Scholar 

  38. Martinet, B.: Régularization d’ inéquations variationelles par approximations sucessives. Rev. Fr. Inform. Rech. Opér. 4, 154–159 (1970)

    MATH  MathSciNet  Google Scholar 

  39. Nedić, A., Ozdaglar, A.: Subgradient methods for saddle point problems. J. Optim. Theory Appl. 142, 205–228 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  40. Nocedal, J., Wright, S.: Numerical Optimization, Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)

    Google Scholar 

  41. Oreifej, O., Li, X., Shah, M.: Simultaneous video stabilization and moving object detection in turbulence. IEEE Trans. Pattern Anal. Mach. Intell. 35, 450–462 (2013)

    Article  Google Scholar 

  42. Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1, 123–231 (2013)

    Google Scholar 

  43. Pati, Y., Kailath, T.: Phase-shifting masks for microlithography: automated design and mask requirements. J. Opt. Soc. Am. A 11, 2438–2452 (1994)

    Article  Google Scholar 

  44. Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2233–2246 (2012)

    Article  Google Scholar 

  45. Powell, M.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, London (1969)

    Google Scholar 

  46. Rockafellar, R.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)

    Article  MATH  MathSciNet  Google Scholar 

  47. Setzer, S., Steidl, G., Tebuber, T.: Deblurring Poissonian images by split Bregman techniques. J. Vis. Commun. Image Represent. 21, 193–199 (2010)

    Article  Google Scholar 

  48. Tao, M., Yuan, X.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  49. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., Knight, K.: Sparsity and smoothness via the fused LASSO. J. R. Stat. Soc. B 67, 91–108 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  50. Tsiaflakis, P., Diehl, M., Moonen, M.: Distributed spectrum management algorithms for multiuser dsl networks. IEEE Trans. Signal Process. 56, 4825–4843 (2008)

    Article  MathSciNet  Google Scholar 

  51. Yang, J.F., Yuan, X.M.: Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization. Math. Comput. 82(281), 301–329 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  52. Yang, J.F., Zhang, Y.: Alternating direction algorithms for L1-problems in compressive sensing. SIAM J. Sci. Comput. 33(1), 250–278 (2011)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

The authors are grateful to Prof. B. S. He, Prof. X. M. Yuan and Prof. C. H. Chen for their valuable comments on the paper. The work of L. S. Hou was supported by 2013NXY43 and the Natural Science Foundation of China (NSFC-11471156). The work of H. J. He was supported by the Natural Science Foundation of China (NSFC-11301123) and the Zhejiang Provincial NSFC Grant (LZ14A010003). The work of J. F. Yang was supported by the Natural Science Foundation of China (NSFC-11371192), the Fundamental Research Funds for the Central Universities (20620140574), the Program for New Century Excellent Talents in University (NCET-12-0252), and the Key Laboratory for Numerical Simulation of Large Scale Complex Systems of Jiangsu Province.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Junfeng Yang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hou, L., He, H. & Yang, J. A partially parallel splitting method for multiple-block separable convex programming with applications to robust PCA. Comput Optim Appl 63, 273–303 (2016). https://doi.org/10.1007/s10589-015-9770-4

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-015-9770-4

Keywords

Navigation