On the Acceleration of Forward-Backward Splitting via an Inexact Newton Method

Abstract

We propose a Forward-Backward Truncated-Newton method (FBTN) for minimizing the sum of two convex functions, one of which smooth. Unlike other proximal Newton methods, our approach does not involve the employment of variable metrics, but is rather based on a reformulation of the original problem as the unconstrained minimization of a continuously differentiable function, the forward-backward envelope (FBE). We introduce a generalized Hessian for the FBE that symmetrizes the generalized Jacobian of the nonlinear system of equations representing the optimality conditions for the problem. This enables the employment of conjugate gradient method (CG) for efficiently solving the resulting (regularized) linear systems, which can be done inexactly. The employment of CG prevents the computation of full (generalized) Jacobians, as it requires only (generalized) directional derivatives. The resulting algorithm is globally (subsequentially) convergent, Q-linearly under an error bound condition, and up to Q-superlinearly and Q-quadratically under regularity assumptions at the possibly non-isolated limit point.

Appendix: Auxiliary Results

Lemma 1

Any proper lsc convex function with nonempty and bounded set of minimizers is level bounded.

Proof

Let h be such function; to avoid trivialities we assume that $\operatorname {{\mathrm {dom}}} h$ is unbounded. Fix $x_\star \in \operatorname *{\mathrm {argmin}} h$ and let R > 0 be such that . Since $\operatorname {{\mathrm {dom}}} h$ is closed, convex, and unbounded, it holds that h attains a minimum on the compact set $\operatorname {bdry} B$, be it m, which is strictly larger than h(x _⋆) (since $\operatorname {dist}( \operatorname *{\mathrm {argmin}} h,\operatorname {bdry} B)>0$ due to compactness of $\operatorname *{\mathrm {argmin}} h$ and openness of B). For x∉B, let $s_x=x_\star +R\tfrac {x-x_\star }{\|x-x_\star \|}$ denote its projection onto $\operatorname {bdry} B$, and let $t_x\mathrel{\mathop:}= \tfrac {\|x-x_\star \|}{R}\geq 1$. Then,

$$\displaystyle \begin{aligned} h(x) {}={} & h\big(x_\star+t_x(s_x-x_\star)) {}\geq{} h(x_\star)+t_x\big(h(s_x)-h(x_\star)\big) {}\geq{} h(x_\star)+t_x\big(m-h(x_\star)\big) \end{aligned} $$

where in the first inequality we used the fact that t _x ≥ 1. Since m − h(x _⋆) > 0 and t _x →∞ as ∥x∥→∞, we conclude that h is coercive, and thus level bounded. □

Lemma 2

Let $H\in \operatorname {S}_+(\mathbb {R}^n)$ with λ _max(H) ≤ 1. Then $H-H^2\in \operatorname {S}_+(\mathbb {R}^n)$ with

(15.122)

Proof

Consider the spectral decomposition for some orthogonal matrix S and diagonal D. Then, where $\tilde D=D-D^2$. Apparently, $\tilde D$ is diagonal, hence the eigenvalues of H − H ² are exactly . The function λ↦λ − λ ² is concave, hence the minimum in $\operatorname {eigs}(\tilde H)$ is attained at one extremum, that is, either at λ = λ _min(H) or λ = λ _max(H), which proves the claim. □

Lemma 3

For any γ ∈ (0, 2∕L _f) the forward-backward operator (15.22) is nonexpansive (in fact, $\frac {2}{4-\gamma L_f}$ -averaged), and the residual is Lipschitz continuous with modulus $\frac {4}{\gamma (4-\gamma L_f)}$.

Proof

By combining [2, Prop. 4.39 and Cor. 18.17] it follows that the gradient descent operator is γL _f∕2-averaged. Moreover, since the proximal mapping is 1∕2-averaged [2, Prop. 12.28] we conclude from [2, Prop. 4.44] that the forward-backward operator T _γ is α-averaged with $\alpha =\frac {2}{4-\gamma L_f}$, thus nonexpansive [2, Rem. 4.34(i)]. Therefore, by definition of α-averagedness there exists a 1-Lipschitz continuous operator S such that $T_\gamma =(1-\alpha )\operatorname {id}+\alpha S$ and consequently the residual is (2α∕γ)-Lipschitz continuous. □