Elsevier

Pattern Recognition

Volume 44, Issue 6, June 2011, Pages 1312-1326
Pattern Recognition

Linearized proximal alternating minimization algorithm for motion deblurring by nonlocal regularization

https://doi.org/10.1016/j.patcog.2010.12.013Get rights and content

Abstract

Non-blind motion deblurring problems are highly ill-posed and so it is quite difficult to find the original sharp and clean image. To handle ill-posedness of the motion deblurring problem, we use nonlocal total variation (abbreviated as TV) regularization approaches. Nonlocal TV can restore periodic textures and local geometric information better than local TV. But, since nonlocal TV requires weighted difference between pixels in the whole image, it demands much more computational resources than local TV. By using the linearization of the fidelity term and the proximal function, our proposed algorithm does not require any inversion of blurring operator and nonlocal operator. Therefore, the proposed algorithm is very efficient for motion deblurring problems. We compare the numerical performance of our proposed algorithm with that of several state-of-the-art algorithms for deblurring problems. Our numerical results show that the proposed method is faster and more robust than state-of-the-art algorithms on motion deblurring problems.

Introduction

The motion deblurring problem is to recover a sharp and clean image from the given blurred image, which is mainly caused by unsteady movements of a camera [1]. Let ΩR2, b be the given blurred image, and k be the blurring kernel. Then we wish to find the unknown true image u:ΩR from the given blurred image and the blurring kernel:b=ku+η,where Ωk(x)dx=1 and k0, η is the Gaussian noise, and is the convolution operator (with some boundary condition). For simplicity, we assume that the blurring kernel k is spatially invariant, i.e., a blurred object looks same regardless of its location in the given image. If the spatially invariant blurring kernel is unknown, then the given problem becomes a blind deconvolution. If the given kernel is known, the problem is a non-blind deconvolution. For the blind deconvolution problem, motion blur is easily estimated by using 1 regularization approach [2], [3] because of the strong sparsity of motion blur. For more details on the blind deconvolution, see [1], [2], [3], [4], [5].

In this paper, we only consider the non-blind motion deblurring problems. Even though, convolution/blurring kernel is already known, it is highly ill-posed problem and so it is quite difficult to find the original sharp and clean image. The reason is obvious since the blurring kernel is a kind of a low pass filter and tends to reduce high frequency parts such as textures and edges. Hence directly inverting kernel without using appropriate regularizer causes highly ringing artifacts around edges and textures of an image. To handle this problem, proper regularization methods are required. The most successful regularizer is the local TV [6] used in deblurring problems [4], [7]. The main advantage of using local TV is that it preserves edges due to its linear penalty on differences between adjacent pixels. But, it tends to flatten inhomogeneous areas, such as textures; see Fig. 1(d). To overcome this shortcoming, nonconvex anisotropic TV regularization techniques [1], [5] based on statistical distribution of the gradient of an image or spatially adaptive TV regularization techniques [8], [9] are used. In this paper, we use nonlocal TV regularization technique [10], [11], [12], [13] to restore periodic texture and edge information of the given blurred and noisy image. Nonlocal TV uses the whole image information instead of using adjacent pixel information. In other words, by averaging the current pixel to the other pixels with similar structure neighborhoods, i.e., patches, we can restore the texture with periodic patterns and the sharp edge from the blurred image; see Fig. 1(c). We note that nonlocal total variation is also an efficient approach for other image restoration problems such as denoising, superresolution, compressive sampling, inpainting, and segmentation [14], [15], [16], [17].

Since nonlocal TV requires weighted difference between pixels in the whole image, it consumes more computational resources than local TV. Hence efficient algorithms for solving nonlocal TV deblurring problems are demanded. Recently, Bregmanized operator splitting algorithm [13] has been proposed to solve nonlocal TV deblurring problems. But this method is not quite efficient, since it uses the split Bregman algorithm for solving the nonlocal TV denoising subproblem for each outer iteration and requires the inverses involving a nonlocal Laplacian operator for inner iterations.

Recently developed several other algorithms, such as the alternating minimization algorithm with a penalty approach [18], the inexact alternating direction method of multipliers [19], and the primal–dual hybrid gradient algorithm [20], for solving image deblurring problems based on local TV regularization, can be applied to solve nonlocal TV deblurring problems without solving nonlocal TV denoising subproblems, i.e., there are no inner iterations. But we note that those algorithms require to compute the inverses involving a blurring operator [18], [20] or a nonlocal Laplacian operator [19].

The purpose of this paper is to develop a new fast algorithm, for solving nonlocal TV deblurring problems, which does not require any inverse of the operators. We adapt a similar framework as for the alternating minimization algorithm proposed by Tseng [21]. As we mentioned above, the current state-of-the-art algorithms, such as the Bregmanized operator splitting algorithm [13], the alternating minimization algorithm with a penalty approach [18], the inexact alternating direction method [19], and the primal–dual hybrid gradient algorithm [20], require to compute the inverses involving a blurring operator or a nonlocal Laplacian operator. This can take much computational resources. Hence this motivates us to develop new algorithms which use a linearization and proximal techniques to overcome the drawback; see Section 3 for details. Our proposed algorithm also does not require inner iterations which are needed for the Bregmanized operator splitting algorithm. The proposed algorithm is faster and more robust than state-of-the-art algorithms [13], [19], [20] for solving nonlocal TV deblurring problems.

In the motion deblurring problem, since we do not know the boundary condition of the given motion blurred image b in Eq. (1), we need to give an appropriate boundary condition on the convolution operator. We consider the reflexive or periodic boundary condition in this paper. For the periodic boundary condition, we use the fast Fourier transform to solve the deblurring problem. The main advantage of FFT is that it only requires O(n log (n)) arithmetic operations for a convolution of the given image of size n, regardless of size of a convolution operator [18], [20]. But the periodic boundary condition is too artificial and so it induces strong boundary artifacts in the restored image. To resolve this problem, various techniques are described in [1], [22], [23], [24]. Among those techniques, we use “edgetaper” function in Matlab for motion deblurring problems with FFT. For the reflexive boundary condition, since the motion blur does not have any specific structure, we cannot use any fast transform-based method [25].

To solve motion deblurring problems, one may consider the following local TV-based variational formulation:minuμΩ|u|+12kub22,where μ>0. As we mentioned earlier, fine periodic structures are not well recovered with this model (see Fig. 1). To overcome this problem, nonlocal TV regularization techniques have been proposed [14], [16], [17]. In the sequel, we use the definitions and notations of the nonlocal functionals introduced in [17] to define nonlocal total variation. Let w:Ω×ΩR+{0} be a symmetric weight function, i.e., w(x,y)=w(y,x). The nonlocal partial derivative at x to y is written as yu(x)(u(y)u(x))w(x,y).The nonlocal gradient operator wu:ΩΩ×Ω is defined as the vector of all partial derivatives at xΩ: wu(x,y)(u(y)u(x))w(x,y),for allyΩ,where w(x,y) is the weight function between x and y defined based on the image u. The nonlocal divergence of a vector ϱ:Ω×ΩR at xΩ can be defined by the adjoint relation with the nonlocal gradient: wu,ϱ=u,divwϱ,which defines the nonlocal divergence as divwϱ(x)Ω(ϱ(x,y)ϱ(y,x))w(x,y)dy.Now, we can define nonlocal total variation functional byTVw(u)=Ω|wu|dx=ΩΩ(u(y)u(x))2w(x,y)dydx.

Before we proceed, we need to define the weight function w(x,y). Let the region Ωw(x)Ω be a neighborhood around xΩ where the weights are positive. The weight function is defined by the solution of the following energy minimization problem [14], [26]:minwE(w),whereE(w)=ΩΩw(x)w(x,y)du2(x,y)2h2+w(x,y)log(w(x,y))w(x,y)dydx,du2(x,y)ΩGa(t)(Fu(xt)Fu(yt))2dt.Ga is the Gaussian kernel with standard deviation a=2, Fu(z)=u(z)B(z), where B(z) is a patch centered at z, and h is the scaling parameter which determines the similarity between different patches. We refer [10] for different types of metric (6) between x and y in Ω. The solution of (4) is the following function:w(x,y)=edu2(x,y)/2h2,xΩ,yΩw(x).

To find true image u in (1), we consider the following formulation:minw,uF(w,u)E(w)+μTVw(u)+12kub22,for the motion deblurring model. In this paper, we use an alternating approach that is considered in [14], [26]:Weightupdate:wk=argminwF(w,uk1),Imageupdate:uk=argminuF(wk,u).We note that Algorithm 1 in Section 4 describes the whole process for non-blind motion deblurring problems in detail.

For notational convenience, we use vector notation, i.e., the 2D M×N image is columnwisely stacked into a vector, for the rest of the paper. Therefore, the unknown true image u is a vector in Rn (n=MN), the observed blurred and noisy image b is a vector in Rn, and the motion blur can be modeled as a large sparse n×n matrix A (with an appropriate boundary condition). Then (1) can be expressed as follows:b=Au+η,where ηN(0,σ2). Therefore, in the sequel, we consider the following formulation for the image update part (9b):minuμwu+12Aub22,where μ>0 and wu=i=1n(wu)i2 with (wu)iRD. We note that D is the size of “neighbor”, where the weight (7) is positive.

The paper is organized as follows. In Section 2, we review several algorithms for solving the TV regularization problem. In Section 3, we describe our proposed algorithm for solving the nonlocal TV deblurring problem. In Section 4, we describe the motion deblurring process and empirically discuss about the permissible range of a stepsize which is crucial for the stability of the proposed algorithm. In Section 5, we report our numerical results on image deblurring problems. In Section 6, we give our conclusions.

Section snippets

Related works

The local TV regularization (2) has been popular ever since its introduction by Rudin et al. [6]. Many researchers have proposed algorithms for solving (2) and its variants. Recently, Goldstein and Osher [27] proposed the split Bregman algorithm for solving (2) by using the Bregman iteration to solve the linear constraint reformulation of it:minu,zμz+12Aub22s.tu=z,and using an alternating approach to approximate the minimization over u and z. Esser et al. [28] proposed a modified version

Linearized proximal alternating minimization algorithm for nonlocal total variation

In this section, we describe our proposed linearized proximal alternating minimization algorithm (abbreviated as LPAMA) and the extended version of the IADM for solving the nonlocal TV motion deblurring problem of the form (11). In our proposed algorithms, we approximate the 2 fidelity term of the problem (11) by a strongly convex quadratic function. In order to obtain this approximation, we use the linearization of the 2 fidelity term and the proximal function as did in the IADM. Then we use

Algorithm framework for motion deblurring and stability

In this section, we give the details of motion deblurring process and analyze the stability of the LPAMA. We also describe how we update the weight and how to choose the parameters for algorithms. First of all, we give the following general algorithm framework for the non-blind motion deblurring problem.

Algorithm 1

Non-blind Motion Deblurring Algorithm Framework

Input: Given blurred and noisy image b and blurring operator A with the periodic or reflexive boundary condition
Initialization: u−1=0 and u0=argmin

Numerical experience on nonlocal TV deblurring problems

All algorithms are implemented with 64bit Matlab (version 7.10). All runs are performed on a laptop with Intel i7-640LM CPU (2.13–2.93 GHz) and 8 GB Memory. The Operation System is 64 bit Linux. We note that we slightly modified the BOSSB (http://www.math.ucla.edu/∼xqzhang/html/code.html) to enhance the performance for the motion deblurring problems, and the IADM and the PDHG algorithms are implemented by us based on the guideline in [19], [20], respectively. In order to speed up the computation

Conclusion

In this paper, we have proposed the linearized proximal alternating minimization algorithm for solving motion deblurring problems based on nonlocal total variation. The linearized proximal alternating minimization algorithm has advantages of avoiding inner loops and the computation of any inverses involving the blurring operator and the nonlocal operator by using the linearization of the fidelity term and the proximal function. We have compared our method with the Bregmanized operator splitting

Acknowledgments

We would like to thank the anonymous reviewers for their valuable suggestions for improving this paper. This work was supported by Priority Research Centers Program (2009-0093827) and Basic Science Research Program (2010-0510-1-3) through the National Research Foundation of Korea funded by the Ministry of Education, Science and Technology.

Sangwoon Yun is a research fellow at School of Computational Sciences in Korea Institute for Advanced Study (KIAS). He received Ph.D. in Mathematics from University of Washington in 2007. He was a research fellow at National University of Singapore, from 2007 to 2009. His research interest is convex and nonsmooth optimization, variational analysis, image processing.

References (40)

  • L. Rudin et al.

    Nonlinear total variation based noise removal algorithms

    Physica D

    (1992)
  • J.P. Oliveira et al.

    Adaptive total variation image deblurring: a majorization-minimization approach

    Signal Process.

    (2009)
  • F. Li et al.

    Variational denoising of partly textured images

    J. Vis. Commun. Image Representation

    (2009)
  • R. Fergus et al.

    Removing camera shake from a single photograph

    ACM Trans. Graphics

    (2006)
  • Q. Shan et al.

    High-quality motion deblurring from a single image

    ACM Trans. Graphics

    (2008)
  • C. Wang et al.

    Multi-scale blind motion deblurring using local minimum

    Inverse Probl.

    (2010)
  • T.F. Chan et al.

    Total variation blind deconvolution

    IEEE Trans. Image Process.

    (1998)
  • A. Levin, Y. Weiss, F. Durand, W.T. Freeman, Understanding and evaluating blind deconvolution algorithms, in: IEEE...
  • G. Gilboa et al.

    Variational denoising of partly textured images by spatially varying constraints

    IEEE Trans. Image Process.

    (2006)
  • G. Gilboa et al.

    Nonlocal linear image regularization and supervised segmentation

    Multiscale Model. Simul.

    (2007)
  • S. Kindermann et al.

    Deblurring and denoising of images by nonlocal functionals

    Multiscale Model. Simul.

    (2005)
  • Y. Lou et al.

    Image recovery via nonlocal operators

    J. Sci. Comput.

    (2010)
  • X. Zhang et al.

    Bregmanized nonlocal regularization for deconvolution and sparse reconstruction

    SIAM J. Imaging Sci.

    (2010)
  • G. Peyre et al.

    Nonlocal regularization of inverse problems

  • X. Bresson, T.F. Chan, Nonlocal unsupervised variational image segmentation models, UCLA Cam-Report,...
  • A. Elmoataz et al.

    Nonlocal discrete regularization on weighted graphs: a framework for image and manifold processing

    IEEE Trans. Image Process.

    (2008)
  • G. Gilboa et al.

    Nonlocal operators with applications to image processing

    Multiscale Model. Simul.

    (2008)
  • Y. Wang et al.

    A new alternating minimization algorithm for total variation image reconstruction

    SIAM J. Imaging Sci.

    (2008)
  • Y. Xiao, J. Yang, A fast algorithm for total variation image reconstruction from random projections, Preprint,...
  • M. Zhu, T. Chan, An efficient primal–dual hybrid gradient algorithm for total variation image restoration, UCLA...
  • Cited by (29)

    • Reaction–diffusion equation based image restoration

      2018, Applied Mathematics and Computation
      Citation Excerpt :

      Aiming at the problem of image quality degradation caused by blurring, many deblurred image restoration algorithms have been proposed, mainly based on partial differential equation (PDE) models, By exploiting these characteristics, ones have proposed many restoration algorithms for deblurring images, such as the literature [17]–[22]. As a type of effective image deblurring algorithms, the partial differential equation (PDE) based models, such as total variation (TV) model [17], nonlocal regularization model [18], Geometrical PDEs [19], and the improved traditional deblurring algorithms, like the improved Richardson–Lucy algorithm [20], regularized structured total least squares model [21] and Fourier-wavelet reg-ularized deconvolution (ForWaRD) algorithm [22], all of these algorithms are playing a very important role in image deblurring. However, the drawback of these algorithms is that the image details can not be well preserved when the image is corrupted by noise and blurring.

    • A blind deconvolution model for scene text detection and recognition in video

      2016, Pattern Recognition
      Citation Excerpt :

      w is the weight and is considered as the Gaussian kernel with a variance of 0.4, which is applied over the L1 norm in gradient space to increase the sharpness of the edges. It works based on the fact that pixels around the edges behave like a Gaussian due to the transition from background to foreground and vice versa [40]. In summary, both λ and w together help to brighten images where there are edges with the R operator.

    • A fast render algorithm of virtual and real motion blur consistence in augmented reality

      2013, Optik
      Citation Excerpt :

      Total variation (TV) algorithm is one of the classical algorithms in restoration of non-blind motion-blurred image. As recent researches, restoration of blur image is used nonlocal total variation regularization methods in Sangwoon and Hyenkyun [14]. The method uses fidelity term linearization and the proximal function, which is not using blur operator and the inverse of nonlocal operator, guaranteeing the validity and effectiveness of algorithms.

    View all citing articles on Scopus

    Sangwoon Yun is a research fellow at School of Computational Sciences in Korea Institute for Advanced Study (KIAS). He received Ph.D. in Mathematics from University of Washington in 2007. He was a research fellow at National University of Singapore, from 2007 to 2009. His research interest is convex and nonsmooth optimization, variational analysis, image processing.

    Hyenkyun Woo is a research fellow at Institute of Mathematical Sciences in Ewha Womans University, Korea. He received the B.S. degree in Electrical Engineering from the Sogang University, Seoul, Korea, in 1996, and the M.S., and Ph.D. degrees in Mathematics from the Yonsei University, Seoul, Korea in 1998, and 2007, respectively. He was with IDIS and Samsung Electronics and developed several video surveillance systems, and image processing algorithms. His research interests include image and video signal processing, and inverse problems.

    View full text