Linearized proximal alternating minimization algorithm for motion deblurring by nonlocal regularization
Introduction
The motion deblurring problem is to recover a sharp and clean image from the given blurred image, which is mainly caused by unsteady movements of a camera [1]. Let , b be the given blurred image, and k be the blurring kernel. Then we wish to find the unknown true image from the given blurred image and the blurring kernel:where and , is the Gaussian noise, and is the convolution operator (with some boundary condition). For simplicity, we assume that the blurring kernel k is spatially invariant, i.e., a blurred object looks same regardless of its location in the given image. If the spatially invariant blurring kernel is unknown, then the given problem becomes a blind deconvolution. If the given kernel is known, the problem is a non-blind deconvolution. For the blind deconvolution problem, motion blur is easily estimated by using regularization approach [2], [3] because of the strong sparsity of motion blur. For more details on the blind deconvolution, see [1], [2], [3], [4], [5].
In this paper, we only consider the non-blind motion deblurring problems. Even though, convolution/blurring kernel is already known, it is highly ill-posed problem and so it is quite difficult to find the original sharp and clean image. The reason is obvious since the blurring kernel is a kind of a low pass filter and tends to reduce high frequency parts such as textures and edges. Hence directly inverting kernel without using appropriate regularizer causes highly ringing artifacts around edges and textures of an image. To handle this problem, proper regularization methods are required. The most successful regularizer is the local TV [6] used in deblurring problems [4], [7]. The main advantage of using local TV is that it preserves edges due to its linear penalty on differences between adjacent pixels. But, it tends to flatten inhomogeneous areas, such as textures; see Fig. 1(d). To overcome this shortcoming, nonconvex anisotropic TV regularization techniques [1], [5] based on statistical distribution of the gradient of an image or spatially adaptive TV regularization techniques [8], [9] are used. In this paper, we use nonlocal TV regularization technique [10], [11], [12], [13] to restore periodic texture and edge information of the given blurred and noisy image. Nonlocal TV uses the whole image information instead of using adjacent pixel information. In other words, by averaging the current pixel to the other pixels with similar structure neighborhoods, i.e., patches, we can restore the texture with periodic patterns and the sharp edge from the blurred image; see Fig. 1(c). We note that nonlocal total variation is also an efficient approach for other image restoration problems such as denoising, superresolution, compressive sampling, inpainting, and segmentation [14], [15], [16], [17].
Since nonlocal TV requires weighted difference between pixels in the whole image, it consumes more computational resources than local TV. Hence efficient algorithms for solving nonlocal TV deblurring problems are demanded. Recently, Bregmanized operator splitting algorithm [13] has been proposed to solve nonlocal TV deblurring problems. But this method is not quite efficient, since it uses the split Bregman algorithm for solving the nonlocal TV denoising subproblem for each outer iteration and requires the inverses involving a nonlocal Laplacian operator for inner iterations.
Recently developed several other algorithms, such as the alternating minimization algorithm with a penalty approach [18], the inexact alternating direction method of multipliers [19], and the primal–dual hybrid gradient algorithm [20], for solving image deblurring problems based on local TV regularization, can be applied to solve nonlocal TV deblurring problems without solving nonlocal TV denoising subproblems, i.e., there are no inner iterations. But we note that those algorithms require to compute the inverses involving a blurring operator [18], [20] or a nonlocal Laplacian operator [19].
The purpose of this paper is to develop a new fast algorithm, for solving nonlocal TV deblurring problems, which does not require any inverse of the operators. We adapt a similar framework as for the alternating minimization algorithm proposed by Tseng [21]. As we mentioned above, the current state-of-the-art algorithms, such as the Bregmanized operator splitting algorithm [13], the alternating minimization algorithm with a penalty approach [18], the inexact alternating direction method [19], and the primal–dual hybrid gradient algorithm [20], require to compute the inverses involving a blurring operator or a nonlocal Laplacian operator. This can take much computational resources. Hence this motivates us to develop new algorithms which use a linearization and proximal techniques to overcome the drawback; see Section 3 for details. Our proposed algorithm also does not require inner iterations which are needed for the Bregmanized operator splitting algorithm. The proposed algorithm is faster and more robust than state-of-the-art algorithms [13], [19], [20] for solving nonlocal TV deblurring problems.
In the motion deblurring problem, since we do not know the boundary condition of the given motion blurred image b in Eq. (1), we need to give an appropriate boundary condition on the convolution operator. We consider the reflexive or periodic boundary condition in this paper. For the periodic boundary condition, we use the fast Fourier transform to solve the deblurring problem. The main advantage of FFT is that it only requires O(n log (n)) arithmetic operations for a convolution of the given image of size n, regardless of size of a convolution operator [18], [20]. But the periodic boundary condition is too artificial and so it induces strong boundary artifacts in the restored image. To resolve this problem, various techniques are described in [1], [22], [23], [24]. Among those techniques, we use “edgetaper” function in Matlab for motion deblurring problems with FFT. For the reflexive boundary condition, since the motion blur does not have any specific structure, we cannot use any fast transform-based method [25].
To solve motion deblurring problems, one may consider the following local TV-based variational formulation:where . As we mentioned earlier, fine periodic structures are not well recovered with this model (see Fig. 1). To overcome this problem, nonlocal TV regularization techniques have been proposed [14], [16], [17]. In the sequel, we use the definitions and notations of the nonlocal functionals introduced in [17] to define nonlocal total variation. Let be a symmetric weight function, i.e., w(x,y)=w(y,x). The nonlocal partial derivative at x to y is written as The nonlocal gradient operator is defined as the vector of all partial derivatives at : where w(x,y) is the weight function between x and y defined based on the image u. The nonlocal divergence of a vector at can be defined by the adjoint relation with the nonlocal gradient: which defines the nonlocal divergence as Now, we can define nonlocal total variation functional by
Before we proceed, we need to define the weight function w(x,y). Let the region be a neighborhood around where the weights are positive. The weight function is defined by the solution of the following energy minimization problem [14], [26]:whereGa is the Gaussian kernel with standard deviation a=2, , where B(z) is a patch centered at z, and h is the scaling parameter which determines the similarity between different patches. We refer [10] for different types of metric (6) between x and y in . The solution of (4) is the following function:
To find true image u in (1), we consider the following formulation:for the motion deblurring model. In this paper, we use an alternating approach that is considered in [14], [26]:We note that Algorithm 1 in Section 4 describes the whole process for non-blind motion deblurring problems in detail.
For notational convenience, we use vector notation, i.e., the 2D image is columnwisely stacked into a vector, for the rest of the paper. Therefore, the unknown true image u is a vector in (n=MN), the observed blurred and noisy image b is a vector in , and the motion blur can be modeled as a large sparse n×n matrix A (with an appropriate boundary condition). Then (1) can be expressed as follows:where . Therefore, in the sequel, we consider the following formulation for the image update part (9b):where and with . We note that D is the size of “neighbor”, where the weight (7) is positive.
The paper is organized as follows. In Section 2, we review several algorithms for solving the TV regularization problem. In Section 3, we describe our proposed algorithm for solving the nonlocal TV deblurring problem. In Section 4, we describe the motion deblurring process and empirically discuss about the permissible range of a stepsize which is crucial for the stability of the proposed algorithm. In Section 5, we report our numerical results on image deblurring problems. In Section 6, we give our conclusions.
Section snippets
Related works
The local TV regularization (2) has been popular ever since its introduction by Rudin et al. [6]. Many researchers have proposed algorithms for solving (2) and its variants. Recently, Goldstein and Osher [27] proposed the split Bregman algorithm for solving (2) by using the Bregman iteration to solve the linear constraint reformulation of it:and using an alternating approach to approximate the minimization over u and z. Esser et al. [28] proposed a modified version
Linearized proximal alternating minimization algorithm for nonlocal total variation
In this section, we describe our proposed linearized proximal alternating minimization algorithm (abbreviated as LPAMA) and the extended version of the IADM for solving the nonlocal TV motion deblurring problem of the form (11). In our proposed algorithms, we approximate the fidelity term of the problem (11) by a strongly convex quadratic function. In order to obtain this approximation, we use the linearization of the fidelity term and the proximal function as did in the IADM. Then we use
Algorithm framework for motion deblurring and stability
In this section, we give the details of motion deblurring process and analyze the stability of the LPAMA. We also describe how we update the weight and how to choose the parameters for algorithms. First of all, we give the following general algorithm framework for the non-blind motion deblurring problem. Algorithm 1 Non-blind Motion Deblurring Algorithm FrameworkInput: Given blurred and noisy image b and blurring operator A with the periodic or reflexive boundary condition Initialization: u−1=0 and
Numerical experience on nonlocal TV deblurring problems
All algorithms are implemented with 64bit Matlab (version 7.10). All runs are performed on a laptop with Intel i7-640LM CPU (2.13–2.93 GHz) and 8 GB Memory. The Operation System is 64 bit Linux. We note that we slightly modified the BOSSB (http://www.math.ucla.edu/∼xqzhang/html/code.html) to enhance the performance for the motion deblurring problems, and the IADM and the PDHG algorithms are implemented by us based on the guideline in [19], [20], respectively. In order to speed up the computation
Conclusion
In this paper, we have proposed the linearized proximal alternating minimization algorithm for solving motion deblurring problems based on nonlocal total variation. The linearized proximal alternating minimization algorithm has advantages of avoiding inner loops and the computation of any inverses involving the blurring operator and the nonlocal operator by using the linearization of the fidelity term and the proximal function. We have compared our method with the Bregmanized operator splitting
Acknowledgments
We would like to thank the anonymous reviewers for their valuable suggestions for improving this paper. This work was supported by Priority Research Centers Program (2009-0093827) and Basic Science Research Program (2010-0510-1-3) through the National Research Foundation of Korea funded by the Ministry of Education, Science and Technology.
Sangwoon Yun is a research fellow at School of Computational Sciences in Korea Institute for Advanced Study (KIAS). He received Ph.D. in Mathematics from University of Washington in 2007. He was a research fellow at National University of Singapore, from 2007 to 2009. His research interest is convex and nonsmooth optimization, variational analysis, image processing.
References (40)
- et al.
Nonlinear total variation based noise removal algorithms
Physica D
(1992) - et al.
Adaptive total variation image deblurring: a majorization-minimization approach
Signal Process.
(2009) - et al.
Variational denoising of partly textured images
J. Vis. Commun. Image Representation
(2009) - et al.
Removing camera shake from a single photograph
ACM Trans. Graphics
(2006) - et al.
High-quality motion deblurring from a single image
ACM Trans. Graphics
(2008) - et al.
Multi-scale blind motion deblurring using local minimum
Inverse Probl.
(2010) - et al.
Total variation blind deconvolution
IEEE Trans. Image Process.
(1998) - A. Levin, Y. Weiss, F. Durand, W.T. Freeman, Understanding and evaluating blind deconvolution algorithms, in: IEEE...
- et al.
Variational denoising of partly textured images by spatially varying constraints
IEEE Trans. Image Process.
(2006) - et al.
Nonlocal linear image regularization and supervised segmentation
Multiscale Model. Simul.
(2007)
Deblurring and denoising of images by nonlocal functionals
Multiscale Model. Simul.
Image recovery via nonlocal operators
J. Sci. Comput.
Bregmanized nonlocal regularization for deconvolution and sparse reconstruction
SIAM J. Imaging Sci.
Nonlocal regularization of inverse problems
Nonlocal discrete regularization on weighted graphs: a framework for image and manifold processing
IEEE Trans. Image Process.
Nonlocal operators with applications to image processing
Multiscale Model. Simul.
A new alternating minimization algorithm for total variation image reconstruction
SIAM J. Imaging Sci.
Cited by (29)
Reaction–diffusion equation based image restoration
2018, Applied Mathematics and ComputationCitation Excerpt :Aiming at the problem of image quality degradation caused by blurring, many deblurred image restoration algorithms have been proposed, mainly based on partial differential equation (PDE) models, By exploiting these characteristics, ones have proposed many restoration algorithms for deblurring images, such as the literature [17]–[22]. As a type of effective image deblurring algorithms, the partial differential equation (PDE) based models, such as total variation (TV) model [17], nonlocal regularization model [18], Geometrical PDEs [19], and the improved traditional deblurring algorithms, like the improved Richardson–Lucy algorithm [20], regularized structured total least squares model [21] and Fourier-wavelet reg-ularized deconvolution (ForWaRD) algorithm [22], all of these algorithms are playing a very important role in image deblurring. However, the drawback of these algorithms is that the image details can not be well preserved when the image is corrupted by noise and blurring.
Learn to model blurry motion via directional similarity and filtering
2018, Pattern RecognitionA blind deconvolution model for scene text detection and recognition in video
2016, Pattern RecognitionCitation Excerpt :w is the weight and is considered as the Gaussian kernel with a variance of 0.4, which is applied over the L1 norm in gradient space to increase the sharpness of the edges. It works based on the fact that pixels around the edges behave like a Gaussian due to the transition from background to foreground and vice versa [40]. In summary, both λ and w together help to brighten images where there are edges with the R operator.
Variational image segmentation model coupled with image restoration achievements
2015, Pattern RecognitionFast linearized alternating direction minimization algorithm with adaptive parameter selection for multiplicative noise removal
2014, Journal of Computational and Applied MathematicsA fast render algorithm of virtual and real motion blur consistence in augmented reality
2013, OptikCitation Excerpt :Total variation (TV) algorithm is one of the classical algorithms in restoration of non-blind motion-blurred image. As recent researches, restoration of blur image is used nonlocal total variation regularization methods in Sangwoon and Hyenkyun [14]. The method uses fidelity term linearization and the proximal function, which is not using blur operator and the inverse of nonlocal operator, guaranteeing the validity and effectiveness of algorithms.
Sangwoon Yun is a research fellow at School of Computational Sciences in Korea Institute for Advanced Study (KIAS). He received Ph.D. in Mathematics from University of Washington in 2007. He was a research fellow at National University of Singapore, from 2007 to 2009. His research interest is convex and nonsmooth optimization, variational analysis, image processing.
Hyenkyun Woo is a research fellow at Institute of Mathematical Sciences in Ewha Womans University, Korea. He received the B.S. degree in Electrical Engineering from the Sogang University, Seoul, Korea, in 1996, and the M.S., and Ph.D. degrees in Mathematics from the Yonsei University, Seoul, Korea in 1998, and 2007, respectively. He was with IDIS and Samsung Electronics and developed several video surveillance systems, and image processing algorithms. His research interests include image and video signal processing, and inverse problems.