A semismooth Newton-CG based dual PPA for matrix spectral norm approximation problems

Abstract

We consider a class of matrix spectral norm approximation problems for finding an affine combination of given matrices having the minimal spectral norm subject to some prescribed linear equality and inequality constraints. These problems arise often in numerical algebra, engineering and other areas, such as finding Chebyshev polynomials of matrices and fastest mixing Markov chain models. Based on classical analysis of proximal point algorithms (PPAs) and recent developments on semismooth analysis of nonseparable spectral operators, we propose a semismooth Newton-CG based dual PPA for solving the matrix norm approximation problems. Furthermore, when the primal constraint nondegeneracy condition holds for the subproblems, our semismooth Newton-CG method is proven to have at least a superlinear convergence rate. We also design efficient implementations for our proposed algorithm to solve a variety of instances and compare its performance with the nowadays popular first order alternating direction method of multipliers (ADMM). The results show that our algorithm, which is warm-started with an initial point obtained from the ADMM, substantially outperforms the pure ADMM, especially for the constrained cases and it is able to solve the problems robustly and efficiently to a relatively high accuracy.

Appendix: Proof of Proposition 4.1

Let $\{Y^i\}_{i\ge 1}$ be a sequence converging to $Y$ such that every element $Y^i\in \mathcal{D}_{\Pi _{\mathcal{B}}}$. This, by Proposition 2.3(i), implies that $\Vert Y^i\Vert _*\ne 1$ for each $i\ge 1$. Let the SVD of $Y^i$ be $Y^i = U^i [\mathrm{Diag}(\sigma ^i)\,\, 0] (V^i)^T$. We consider the following three cases.

• (i) $\Vert Y\Vert _*<1$. In this case, $\Pi _\mathcal{B}(\cdot )$ is continuously differentiable at $Y$ and its generalized Jacobian is a singleton consisting of the identity operator from $\mathfrak {R}^{m\times n}$ to itself.

• (ii) $\Vert Y\Vert _*=1$. Since $Y$ can be approximated by a sequence in the interior of $\mathcal{B}$, it follows that the identity operator $\mathcal{I}$ is always an element of $\partial _{B}\Pi _\mathcal{B}(Y)$. To obtain the remaining elements, we consider the case in which $\{Y^i\}$ has an infinite subsequence outside $\mathcal{B}$. Without loss of generality, we assume that $\Vert Y^i\Vert _* >1$ for all $i$. By passing to a subsequence if necessary, we know that there exists a positive integer $N\in [r,\,m]$ such that $N= k_1(\sigma ^i)$ for each $i$. Therefore, one has

  $$\begin{aligned} g^i _k:=(\Pi _{\mathbb {B}}(\sigma ^i))_k \; =\; \left\{ \begin{array}{l@{\quad }l} \sigma _k^i- \frac{1}{N}\Big (\sum _{j=1}^N \sigma ^i_j -1\Big ),&{} \quad 1\le k\le N, \\ 0,&{}\quad \mathrm{otherwise} \end{array} \right. \end{aligned}$$

  and

  $$\begin{aligned} \Pi _{\mathbb {B}}^\prime (\sigma ^i)= \left[ \begin{array}{c@{\quad }c} I_N&{}\quad 0\\ 0&{}\quad 0 \end{array} \right] -\frac{1}{N} \left[ \begin{array}{cc} \mathbf 1 _{N\times N}&{}\quad 0\\ 0&{}\quad 0 \end{array} \right] . \end{aligned}$$

  For each $i$, it holds that

  $$\begin{aligned} \big (\Omega (\sigma ^i)\big )_{kj}&= \left\{ \begin{array}{cl} 1,&{}\quad \mathrm{if}\,\, k,j\in \alpha _1\cup \alpha _2, \\ {g_k^i\over \sigma _k^i- \sigma _{j}^i}, &{}\quad \mathrm{if}\,\, k\in \alpha _1\cup \alpha _2,\,j\in \alpha _3,\\ {g_j^i\over \sigma _j^i- \sigma _{k}^i}, &{}\quad \mathrm{if}\,\, k\in \alpha _3,\, j\in \alpha _1 \cup \alpha _2,\,\\ 0,&{}\quad \mathrm{if}\,\, k, \,j\in \alpha _3, \end{array} \right. \\ \big (\Gamma (\sigma ^i)\big )_{kj}&= \left\{ \begin{array}{cl} {g_k^i + g_k^j\over \sigma _k^i+ \sigma _{j}^i},&{}\quad \mathrm{if}\,\, k,j \in \alpha _1\cup \alpha _2, \\ {g_k^i\over \sigma _k^i+ \sigma _{j}^i}, &{}\quad \mathrm{if}\,\, k\in \alpha _1\cup \alpha _2,\,j\in \alpha _3,\\ {g_j^i\over \sigma _k^i+ \sigma _{j}^i}, &{}\quad \mathrm{if}\,\, k\in \alpha _3,\,j\in \alpha _1\cup \alpha _2,\\ 0,&{}\quad \mathrm{if}\,\, k,\,j\in \alpha _3 \end{array} \right. \end{aligned}$$

  and

  $$\begin{aligned} \big (\mathcal{F}(\sigma ^i)\big )_{kj} =\left\{ \begin{array}{cl} -{1\over N},&{}\quad \mathrm{if}\,\, k,j\in \alpha _1\cup \alpha _2, \\ 0,&{}\quad \mathrm{otherwise}, \end{array} \right. \big (\Upsilon (\sigma ^i)\big )_{k} =\left\{ \begin{array}{cl} {g_k^i\over \sigma _k^i },&{}\quad \mathrm{if}\,\, k\in \alpha _1\cup \alpha _2, \\ 0,&{}\quad \mathrm{if}\,\, k\in \alpha _3. \end{array} \right. \end{aligned}$$

  Now from Proposition 2.3(i), we know that for any given $H\in \mathfrak {R}^{m\times n}$,

  $$\begin{aligned} \Pi '_{\mathcal{B}}(Y^i)H = U^i\left[ W^i- \frac{\mathrm{Tr}(\widetilde{H}_{11}^i)}{N}\left[ \begin{array}{cc} I_N&{}0\\ 0&{} 0 \end{array} \right] , \,\, \mathrm{Diag}\big (\Upsilon (\sigma ^i)\big )\widetilde{H}_2^i \right] (V^i)^T, \end{aligned}$$

  (69)

  where the matrix $W^i\in \mathfrak {R}^{m\times m}$ is defined by

  $$\begin{aligned} W^i=\Omega (\sigma ^i) \circ S(\widetilde{H}_1^i)+ \Gamma (\sigma ^i)\circ T(\widetilde{H}_1^i) \end{aligned}$$

  with $\widetilde{H}_1^i\in \mathfrak {R}^{m\times m}, \widetilde{H}_2^i\in \mathfrak {R}^{m\times (n-m)}$, $[\widetilde{H}_1^i\,\,\widetilde{H}_2^i]= (U^i)^THV^i$ and $\widetilde{H}_{11}^i$ being the matrix extracted from the first $N$ columns and rows of $\widetilde{H}_1^i$. By simple algebraic computations, we have

  $$\begin{aligned}&\lim _{i\rightarrow \infty } \big (\Omega (\sigma ^i)\big )_{\alpha _1\alpha _3} = \lim _{i\rightarrow \infty } \big (\Omega (\sigma ^i)\big )_{\alpha _3\alpha _1}^T = \mathbf 1 _{r\times (m-N)},\\&\lim _{i\rightarrow \infty } \big (\Gamma (\sigma ^i)\big )_{\alpha _1(\alpha _1\cup \alpha _2\cup \alpha _3)} = \big (\Gamma (\sigma ^i)\big )_{(\alpha _1\cup \alpha _2\cup \alpha _3)\alpha _1}^T = \mathbf 1 _{r\times m},\\&\lim _{i\rightarrow \infty } \big (\Upsilon (\sigma ^i)\big )_{\alpha _1} = \mathbf 1 _r. \end{aligned}$$

  Note that $\left\{ \Big (\big (\Omega (\sigma ^i)\big )_{\alpha _2\alpha _3},\, \big (\Gamma (\sigma ^i)\big )_{\alpha _2\alpha _2},\,\big (\Gamma (\sigma ^i)\big )_{\alpha _2\alpha _3},\, \big (\Upsilon (\sigma ^i)\big )_{\alpha _2}\Big )\right\} _{i\ge 1}$ is bounded and $\mathcal{S}_N$ is the set of cluster points associated with the sequence. By taking limits on both sides of (69), we are able to establish the conclusion that for any $\mathcal{V} (\ne \mathcal{I})\in \partial _{B} \Pi _{\mathcal{B}}(Y)$, there exist an integer $N\in [r,\,m]$, $(\Omega _{\alpha _2\alpha _3}^\infty , \Gamma _{\alpha _2\alpha _2}^\infty ,\Gamma _{\alpha _2\alpha _3}^\infty ,\Upsilon _{\alpha _2}^\infty )\in \mathcal{S}_N$ and singular vector matrices $U^\infty ,V^\infty$ of $Y$ such that for any $H\in \mathfrak {R}^{m\times n}$, (33) is valid.

• (iii) $\Vert Y\Vert _*>1$. Taking a subsequence if necessary, we know that there exists a positive integer $N\in [k_1(\sigma ),k_2(\sigma )]$ such that $N=k_1(\sigma ^i)$ for each $i$. Therefore,

  $$\begin{aligned} g^i_k:=(\Pi _{\mathbb {B}}(\sigma ^i)_k= \left\{ \begin{array}{ll} \sigma _k^i - \frac{1}{N}\Big ( \sum _{j=1}^N \sigma _j^i-1\Big ),&{} \quad 1\le k \le N, \\ 0,&{} \quad \mathrm{otherwise} \end{array} \right. \end{aligned}$$

  and

  $$\begin{aligned} \Pi _{\mathbb {B}}'(\sigma ^i)= \left[ \begin{array}{cc} I_N&{}0\\ 0&{} 0 \end{array} \right] -{1\over N} \left[ \begin{array}{cc} \mathbf 1 _{N\times N}&{}0\\ 0&{} 0 \end{array} \right] . \end{aligned}$$

  For each $i$, it holds that

  $$\begin{aligned} \big (\Omega (\sigma ^i)\big )_{kj}&= \left\{ \begin{array}{cl} 1,&{}\quad \mathrm{if}\,\, k,j\in \gamma _1, \\ {g_k^i\over \sigma _k^i- \sigma _{j}^i}, &{}\quad \mathrm{if}\,\, k\in \gamma _1,\,j\in \gamma _2,\\ {g_j^i\over \sigma _j^i- \sigma _{k}^i}, &{}\quad \mathrm{if}\,\, k\in \gamma _2,\,j\in \gamma _1,\\ 0&{}\quad \mathrm{if}\,\, k,\,j\in \gamma _2, \end{array} \right. \\ \big (\Gamma (\sigma ^i)\big )_{kj}&= \left\{ \begin{array}{cl} {g_k^i + g_j^i\over \sigma _k^i+ \sigma _{j}^i},&{}\quad \mathrm{if}\,\, k,j \in \gamma _1, \\ {g_k^i\over \sigma _k^i+ \sigma _{j}^i}, &{}\quad \mathrm{if}\,\, k\in \gamma _1,\,j\in \gamma _2,\\ {g_j^i\over \sigma _k^i+ \sigma _{j}^i}, &{}\quad \mathrm{if}\,\, k\in \gamma _2,\,j\in \gamma _1,\\ 0,&{}\quad \mathrm{if}\,\, k,\,j \in \gamma _2 \end{array} \right. \end{aligned}$$

  and

  $$\begin{aligned} \big (\mathcal{F}(\sigma ^i)\big )_{kj} =\left\{ \begin{array}{cl} -{1\over N},&{}\quad \mathrm{if}\,\, k,j\in \gamma _1, \\ 0,&{}\quad \mathrm{otherwise}, \end{array} \right. \quad \big (\Upsilon (\sigma ^i)\big )_{k} =\left\{ \begin{array}{cl} {g_k^i\over \sigma _k^i },&{}\quad \mathrm{if}\,\, k\in \gamma _1, \\ 0,&{}\quad \mathrm{otherwise}. \end{array} \right. \end{aligned}$$

  Then the equality (69) is also valid. Simple calculations show that

  $$\begin{aligned}&\lim \limits _{i\rightarrow \infty } \big (\Omega (\sigma ^i)\big )_{\beta _1\gamma _2}= \lim \limits _{i\rightarrow \infty } \big (\Omega (\sigma ^i)\big )_{\gamma _2\beta _1}^T = \Omega _{\beta _1\gamma _2}, \\&\lim \limits _{i\rightarrow \infty } \big (\Omega (\sigma ^i)\big )_{\beta _2\beta _4}= \lim \limits _{i\rightarrow \infty } \big (\Omega (\sigma ^i)\big )_{\beta _4\beta _2}^T = \Omega _{\beta _2\beta _4}, \\&\lim \limits _{i\rightarrow \infty } \big (\Gamma (\sigma ^i)\big )_{\gamma _1(\gamma _1 \cup \gamma _2)}= \lim \limits _{i\rightarrow \infty } \big (\Gamma (\sigma ^i)\big )_{(\gamma _1\cup \gamma _2)\gamma _1}^T = \Gamma _{\gamma _1({\gamma _1\cup \gamma _2})}, \\&\lim \limits _{i\rightarrow \infty } \big (\Upsilon (\sigma ^i)\big )_{\gamma _1}= \Upsilon _{\gamma _1}. \end{aligned}$$

  Note that $\left\{ \big (\Omega (\sigma ^i)\big )_{\beta _2\beta _3}\right\}$ is bounded and $\mathcal{S}_N$ is the set of cluster points associated with the sequence. By taking limits on both sides of (69), we have the conclusion that for any $\mathcal{V}\in \partial _{B} \Pi _{\mathcal{B}}(Y)$, there exist an integer $N\in [k_1(\sigma ),\,k_2(\sigma )]$, $\Omega _{\beta _2\beta _3}^\infty \in \mathcal{S}_N$ and singular vector matrices $U^\infty ,V^\infty$ of $Y$ such that for any $H\in \mathfrak {R}^{m\times n}$, (34) holds. This completes the proof.