Accelerated Alternating Descent Methods for Dykstra-Like Problems

Abstract

This paper extends recent results by the first author and T. Pock (ICG, TU Graz, Austria) on the acceleration of alternating minimization techniques for quadratic plus nonsmooth objectives depending on two variables. We discuss here the strongly convex situation, and how ‘fast’ methods can be derived by adapting the overrelaxation strategy of Nesterov for projected gradient descent. We also investigate slightly more general alternating descent methods, where several descent steps in each variable are alternatively performed.

Appendix: An Approximation Result

In this appendix, we show that although this is not totally obvious at first glance, the discrete energy $J_\varepsilon (\mathbf {u})$ is an approximation of the isotropic total variation. The result is more precisely as follows. To simplify we work in the domain $\Omega =(0,1)^2$ (extension to more general regular domains is not difficult) and we define, for $N\ge 1$ an integer, the functional, defined for $u\in L^1(\Omega )$,

$$\begin{aligned} \mathcal {F}_{\varepsilon ,N}(u) = {\left\{ \begin{array}{ll} \frac{1}{N}J_{\varepsilon /N}^{N,N}(\mathbf {u}) &{}\quad \text { if } \mathbf {u}=(u_{i,j})_{1\le i,j\le N}, u(x)= \sum \nolimits _{i=1}^N\sum \nolimits _{j=1}^N u_{i,j} \chi _{(\frac{i-1}{N},\frac{j}{N})\times (\frac{j-1}{N},\frac{j}{N})}(x) \text { a.e.,} \\ +\infty &{}\quad \text { else.} \end{array}\right. } \end{aligned}$$

here, $J_{\varepsilon /N}^{N,N}$ is a notation for the energy (44) in case $m=n=N$ (and with the smoothing parameter $\varepsilon /N$). We also denote $\Phi _\varepsilon (p):= |p|^2/(2\varepsilon )$ if $|p|\le \varepsilon$, $|p|-\varepsilon /2$ else and recall that for $u\in BV(\Omega )$ a function with bounded variation $|Du|(\Omega )<+\infty$ [17, 22], $\int _\Omega \Phi _\varepsilon (Du)= \int _\Omega \Phi _\varepsilon (\nabla u)\hbox {d}x + |D^su|$ where $Du=\nabla u \hbox {d}x +D^su$ is the Radon-Nikodym decomposition of Du as an absolutely continuous and singular part, see [15]. We introduce the functional

$$\begin{aligned} \mathcal {F}_\varepsilon (u) = {\left\{ \begin{array}{ll} \Phi _\varepsilon (Du)(\Omega ) &{}\quad \text {if } u\in BV(\Omega ),\\ +\infty &{}\quad \text {if } u\in L^1(\Omega )\setminus BV(\Omega ). \end{array}\right. } \end{aligned}$$

Then, one can show that $\mathcal {F}_\varepsilon$ can also be defined by duality, as follows:

$$\begin{aligned} \mathcal {F}_\varepsilon (u)= & {} \sup \left\{ \int _\Omega u(x)\text {div}\,\varphi (x)\hbox {d}x - \frac{\varepsilon }{2}\int _\Omega |\varphi (x)|^2 \hbox {d}x \,:\,\right. \nonumber \\&\left. \varphi \in C_c^\infty (\Omega ;\mathbb {R}^2), |\varphi (x)|\le 1\quad \forall x\in \Omega \right\} \end{aligned}$$

(47)

One has the following result:

Theorem 4

As $N\rightarrow \infty$, $\mathcal {F}_{\varepsilon ,N}$ $\Gamma$-converges to $\mathcal {F}_\varepsilon$. Moreover, if for some sequence $(u^N)\in L^1(\Omega )^{\mathbb {N}}$, $\mathcal {F}_{\varepsilon ,N}(u^N)\le C<+\infty$, then there exists $u\in BV(\Omega )$, a subsequence $(u^{N_k})_k$ and a sequence of constants $(a_k)_k$ such that that $u^{N_k}-a_k\rightarrow u$ in $L^1(\Omega )$.

For the proper definition and main properties of $\Gamma$-convergence, see for instance [6, 14]. The theorem establishes that images minimizing $J_{\varepsilon /N}^{N,N}$ ($+$ other terms such as a quadratic penalization) should be close if N is large to minimizers of the isotropic ‘Huber-total variation’ $\mathcal {F}_\varepsilon$, in the continuum. The proof is easy, however not really found in this form in the literature, as far as we know. The closest results are maybe the $\Gamma$-convergence theorems of Cai et al. [7] in the context of wavelet-based approximations of the total variation.

Proof

It is enough to prove: (i) that if $u^N\in L^1(\Omega )$ is such that $\ell =\liminf _N \mathcal {F}_{\varepsilon ,N}(u^N)<\infty$, then not only one can extract $u^{N_k}$ which converges to some u, but in addition $\mathcal {F}_\varepsilon (u)\le \ell$; (ii) that given u with finite total variation, one can build a sequence $u^N$ with $\limsup _N \mathcal {F}_{\varepsilon ,N}(u^N)\le \mathcal {F}_\varepsilon (u)$.

For point (i), we first consider a subsequence $(u^{N_k})$ such that $\ell = \lim _k \mathcal {F}_{\varepsilon ,N_k}(u^{N_k})$. Then, we see that since for all k (large enough) $\mathcal {F}_{\varepsilon ,N_k}(u^{N_k})<+\infty$, by definition $u^{N_k}$ is piecewise constant and can be written

$$\begin{aligned} u^{N_k}(x)= \sum _{i=1}^{N_k}\sum _{j=1}^N u^k_{i,j} \chi _{\left( \frac{i-1}{N_k},\frac{j}{N_k}\right) \times \left( \frac{j-1}{N_k},\frac{j}{N_k}\right) }(x) \end{aligned}$$

for some matrix $\mathbf {u}^k=(u^k_{i,j})_{1\le i,j\le N_k}$. Then we observe that for some constant $\sigma >0$,

$$\begin{aligned}&\mathcal {F}_{\varepsilon ,N_k}(u^{N_k}) +\frac{\varepsilon }{2} \ge \mathcal {F}_{0,N_k}(u^{N_k})\\&\qquad \ge \sigma \frac{1}{N_k}\sum _{i,j} \left( |u^{k}_{i+1,j}-u^{k}_{i,j}| +|u^{k}_{i,j+1}-u^{k}_{i,j}|\right) \\&\quad = \sigma |Du^{N_k}|(\Omega ). \end{aligned}$$

Hence $|Du^{N_k}|(\Omega )$ is bounded, showing that $(u^{N_k}-a_k)_k$ is precompact in $L^1(\Omega )$, where $a_k$ is the average of the function $u^{N_k}$ in $\Omega$. Without loss of generality, we assume $a_k=0$ and we denote by u the limit of a subsequence (which for convenience we do not relabel). We must now show that $\mathcal {F}_\varepsilon (u)\le \ell$.

Let $\delta >0$, and let $\varphi =(\varphi ^1,\varphi ^2)\in C_c^\infty (\Omega ;\mathbb {R}^2)$ be a smooth vector field with $|\varphi (x)|^2=\varphi ^1(x)^2+\varphi ^2(x)^2\le 1-\delta$ for all $x\in \Omega$. Observe that

$$\begin{aligned} \int _\Omega u^{N_k}(x)\text {div}\,\varphi (x)\hbox {d}x= & {} \sum _{i,j} u^k_{i,j} \int _{\left( \frac{i-1}{N_k},\frac{j}{N_k}\right) \times \left( \frac{j-1}{N_k},\frac{j}{N_k}\right) }\text {div}\,\varphi (x)\hbox {d}x\\= & {} \sum _{i,j} \left( u^k_{i+1,j}-u^k_{i,j}\right) \varphi ^1_{i+\frac{1}{2},j} \\&+\, \left( u^k_{i,j+1}-u^k_{i,j}\right) \varphi ^2_{i,j+\frac{1}{2}} \end{aligned}$$

where $\varphi ^1_{i+\frac{1}{2},j}$ is the flux of $\varphi$ through the vertical segment $\{\frac{i}{N_k}\}\times (\frac{j-1}{N_k},\frac{j}{N_k})$ and $\varphi ^2_{i,j+\frac{1}{2}}$ is the flux through the horizontal segment $(\frac{i-1}{N_k},\frac{j}{N_k})\times \{\frac{j}{N_k}\}$.

Assume (i, j) are both odd or even. Denote by ${\bar{x}}= (i/N_k,j/N_k)$: as $\varphi$ is smooth, one clearly has that $N_k\varphi ^1_{i+1/2,j} = \varphi ^1({\bar{x}})+ O(1/N_k)$, etc., and, in fact,

$$\begin{aligned}&N_k^2{\mathcal {N}}_{i,j}^2:= \left( N_k\varphi ^1_{i+\frac{1}{2},j}\right) ^2 +\left( N_k\varphi ^2_{i,j+\frac{1}{2}}\right) ^2\\&\qquad +\,\left( N_k\varphi ^1_{i+\frac{1}{2},j+1}\right) ^2+ \left( N_k\varphi ^2_{i+1,j+\frac{1}{2}}\right) ^2\\&\quad \le 2(1-\delta ) + O\left( \frac{1}{N_k^2}\right) \le 2 \end{aligned}$$

if $N_k$ is large enough. As a consequence

$$\begin{aligned}&\left( u^k_{i+1,j}-u^k_{i,j}\right) \varphi ^1_{i+\frac{1}{2},j} \nonumber \\&\qquad +\, \left( u^k_{i,j+1}-u^k_{i,j}\right) \varphi ^2_{i,j+\frac{1}{2}}\nonumber \\&\qquad +\, \left( u^k_{i+1,j+1}-u^k_{i,j+1}\right) \varphi ^1_{i+\frac{1}{2},j+1} \nonumber \\&\qquad +\, \left( u^k_{i+1,j+1}-u^k_{i+1,j}\right) \varphi ^2_{i+1,j+\frac{1}{2}} - \frac{\varepsilon }{2}\mathcal {N}_{i,j}^2\nonumber \\&\quad \le \frac{1}{N_k} TV^{4,\varepsilon /N_k}_{i,j}\left( \mathbf {u}^k\right) . \end{aligned}$$

(48)

Thanks to the smoothness of $\varphi$, one can check easily that

$$\begin{aligned} \sum _{(i,j)\text { even}} {\mathcal {N}}_{i,j}^2 + \sum _{(i,j)\text { odd}} {\mathcal {N}}_{i,j}^2 \rightarrow \int _\Omega |\varphi (x)|^2 \hbox {d}x \end{aligned}$$

as $k\rightarrow \infty$, hence, summing (48) over all (i, j) both odd or both even, we find (using also the fact that $\varphi$ has compact support) that

$$\begin{aligned}&\int _\Omega u^{N_k}(x)\text {div}\,\varphi (x)\hbox {d}x - \frac{\varepsilon }{2}\int _\Omega |\varphi (x)|^2 \hbox {d}x\\&\quad +\,o(1)\le \frac{1}{N_k}J^{N_k,N_k}_{\varepsilon /N_k}\left( \mathbf {u}^k\right) = \mathcal {F}_{\varepsilon ,N_k}\left( u^{N_k}\right) . \end{aligned}$$

In the limit, we find that

$$\begin{aligned} \int _\Omega u(x)\text {div}\,\varphi (x)\hbox {d}x - \frac{\varepsilon }{2}\int _\Omega |\varphi (x)|^2 \hbox {d}x \le \ell . \end{aligned}$$

Thanks to (47), we deduce that $\mathcal {F}_\varepsilon (u)\le \ell$.

We now must prove (ii). We only sketch the proof, which is very simple: one first observes that as any $u\in BV(\Omega )$ can be approximated by a sequence $(u_n)$ with $u_n\in C^\infty (\overline{\Omega })$, $u_n\rightarrow u$ in $L^1(\Omega )$ and $\int _\Omega \Phi _\varepsilon (\nabla u_n(x))\hbox {d}x=\mathcal {F}_\varepsilon (u)$, it is enough to show the result for a smooth function and use then a diagonal argument.

But if u is smooth, letting simply for each N, $u^N_{i,j}=u((i-1/2)/N,(j-1/2)/N)$, one first observes that

$$\begin{aligned} u^N(x):=\sum _{i,j} u^N_{i,j}\chi _{\left( \frac{i-1}{N},\frac{j}{N}\right) \times \left( \frac{j-1}{N},\frac{j}{N}\right) }(x) \rightarrow u(x) \end{aligned}$$

uniformly in $\Omega$, and then that $\mathcal {F}_{\varepsilon ,N}(u^N)$ is a finite-difference approximation of $\int _\Omega \Phi _\varepsilon (u(x))\hbox {d}x$, which converges to this limit as $N\rightarrow \infty$. $\square$