Improved structure tensor for fine-grained texture inpainting
Introduction
Digital image inpainting entails automatically repairing lost or deteriorated parts of images or removing unwanted elements from pictures in an unobservable way. Smart inpainting techniques have broad applications in computer vision and image processing, such as the restoration of aged paintings or damaged photos for conservation purposes [1], text or object removal from photographs for special effects [2], disocclusion in computer vision [[3], [4]], error concealment [5], and zooming and super resolution [[6], [7]].
Texture is a common visual phenomenon that contains important information about the spatial arrangement of color or intensities in an image or a selected region of an image. An image texture is generally recognized to be a set of metrics calculated in image processing and designed to quantify the perceived texture of an image [8], or referred to as visual elements or primitive texels that are spatially repeated deterministically or stochastically [9]. Although it is quite easy for human observers to perceptually recognize and describe a texture in empirical terms, there is no general agreement of a precise definition as it can be formulated based on a particular application. For textured images containing undesirable artifacts, deteriorated textures would inevitably degrade the overall visual quality of the image and would urgently need to be repaired; hence, texture inpainting is an essential part of digital image processing, although it remains quite challenging work because of the lack of precise texture formulation.
Assume a standard image model as where is the original image, is a linear observation operator that is the damaged operator in the inpainting case, is the additive noise; thus, is the observed image. Let be the whole image domain and be the domain to be inpainted with its boundary ; hence , where is a rectangle of size pixels.
In essence, restoring damaged images is equivalent to inferring the missing information in by leveraging the remaining information in in either a patch-wise or a pixel-wise evolution process, necessarily conducted by using some structural features extracted from the known areas. For fine-grained textured images, e.g., the image depicted in Fig. 1 b—some fine-grained details are missing because of text overlap. We find that such subtle and structurally dense detailed features are difficult to effectively extract by the derivative-based feature detectors of many inpainting methods, such as the gradient magnitude in the TV model [10] and the gradient-based structure tensor (ST) [11]. This is attributed to the insensitivity of the first-order derivative to fine and fractal-like image details [12]; therefore, the structural features acquired by these existing detecting operators are essentially insufficient to guide the evolution process, which may easily lead to the detail-blurring phenomenon (Fig. 1 c in the newly inpainted regions). However, with the help of a medical microscope, a surgeon, who is even unable to see complex human organs clearly, is capable of performing a delicate operation. In this paper, we demonstrate the construction of a new feature detector that plays the role of such a microscope to elegantly conduct the inpainting process. Our texture-inpainting model consists of two steps: we firstly construct a novel feature detector specially designed for fine-grained textures, and then build an anisotropic PDE model to restore damaged texture images by integrating the proposed detector.
A large number of research projects utilize patch-wise exemplar-based methods to synthesize texture to complete damaged texture images. For example, Criminisi et al. [13] present an exemplar-based completion method that cheaply generates new patches by sampling and copying two-dimensional patterns from the source area to propagate the linear structure and texture information. The main idea is the design of a hole filling order by using the angle between the isophote direction and the normal direction of the local boundary. On this basis, Zhang et al. [14] and Sangeetha et al. [15] carry out image completion based on their respective improved exemplar-based texture synthesis techniques, the former introducing a method to compute the main direction of the texture and completing the image by limiting the search to one direction to raise the completion speed, and the latter computing the filling order and search direction by leveraging the tensors of texture. Recently, Alotaibi et al. [16] presented a new image completion method, which is based on first creating image structures (regions boundaries) in the hole and then propagating texture from surrounding areas constrained by this structure. These completion methods are highly effective in reproducing textures without a blurry appearance. However, they could easily lead to undesired dislocation deficiencies, i.e., fractured textures, at reconstructed structures or between quilted texture patches due to a patch-wise propagation process, which inevitably results in discontinuities of margins and an unnatural transition of textures when holes are filled.
Moreover, structural propagation techniques, which have been widely reported in the literature on image inpainting [[17], [18], [19], [20], [10]], fill in small-sized gaps or undesired areas in images by smoothly propagating surrounding structures inside the areas by considering their angle of arrival and possible curvature, and are thus capable of avoiding dislocation artifacts due to the pixel-wise diffusion process. Bertalmio et al. pioneered a novel inpainting scheme based on a high-order nonlinear Partial Differential Equation (PDE) model [17]. Inspired by their work, subsequent geometry and variational PDE-based inpainting models [[18], [19], [20], [10]] were developed. Typically, Chan et al. [18] obtained three new inpainting models based on total variation (TV) minimization [21], Mumford–Shah (MS) regularity [22], and curvature driven diffusion (CDD) [23], respectively. Zhang et al. [24] analyzed the physical characteristics of the TV model and p-Laplace operator in local coordinates and proposed a new p-Laplace-based inpainting model to overcome the staircase effect. The nonlinear PDE of their model is where and denote the first-order gradient and divergence, respectively, and is its conductivity coefficient. The PDE (2) is essentially the TV model when , whereas it is equivalent to the Laplace model when .
Tai et al. [25] suggested an inpainting algorithm that propagated the information into the inpainting domain along the isophote direction by solving the TV-Stokes equation. All these PDEs utilize the absolute value of the first-order gradient as an edge detector, allowing simultaneous inpainting of smooth areas and sharp margins without giving rise to dislocation deficiencies. However, this edge detector is quite insensitive to textures, and might even regard high-frequency fine-grained textures as noise that will be smoothed out during the diffusion process. The diffusion mechanism guided by this edge detector inevitably results in blurry deficiencies on the reconstructed texture areas. As a consequence, these PDE models have been limited to non-texture regions and are thus unable to obtain satisfactory inpainted results on damaged texture images.
More recently, tensor diffusion filtering [[11], [26], [27], [28]] has been broadly applied to image processing. In these models, the diffusion coefficient, instead of being a scalar value, is a function of the matrix-valued Structure Tensor (ST) [26], which is essentially the Cartesian product of the first-order gradient vector with itself, and could accurately estimate the orientation and magnitude of structures in local areas. Compared with the above-mentioned PDE models, tensor diffusion is a truly structure-adaptive and anisotropic filtering process.
As our work aims to achieve effective texture processing, we attempt to introduce tensor diffusion to image inpainting by inserting ST into an anisotropic PDE to repair broken structures as well as textures. However, the classic ST has a few limitations when processing fine-grained textures. First, although the integer-order differential is a simple and common mathematical scheme, e.g., the classical gradient in ST [28], tensor diffusion based on such a scheme cannot process the finer details satisfactorily, and may lead to awkward problems, such as edge blurring and a loss of affine details in structurally dense and fine-grained texture regions. Second, since the original texture images are uniformly sampled at the Nyquist rate according to Shannon’s sampling theorem, spectrum aliasing may occur during the calculation of the Cartesian product due to an inadequate sampling rate; consequently, the loss of some fine-grained features is inevitable. Third, pixels sharing the same tensor characteristics are often not located close together, such as those located at the same position in different texture primitive patterns, or along the same edges [[29], [30], [31], [32]]. In view of the characteristic of repeatability of textures, the use of a non-local filtering scheme to regularize the obtained tensor data contributes to guide the propagation process to facilitate inferring when damaged textures are filled.
The goal of our article is to demonstrate in detail a method modifying classic ST to overcome these limitations to subtly restore damaged fine-scale textures free of dislocation and blurring deficiencies. Since PDE-based inpainting methods are capable of effectively recovering image structures, we expand this to texture inpainting by incorporating the modified ST to perform near state-of-the-art anisotropic diffusion. To the best of our knowledge, inpainting of fine-grained texture from this viewpoint has not been considered before, and a large number of experimental results demonstrate that this attempt is reasonable and feasible.
Our contributions are thus twofold. The first and foremost contribution of this work is to improve the traditional ST with three modifications to extract subtle texture features, i.e.,
i. Compared with an integer-order differential, a fractional differential is a more powerful mathematical method for processing complex and fractal-like texture details [[12], [33], [34], [35]]. To efficiently process this image component in the context of fine-grained texture inpainting, we propose a novel Fractional Structure Tensor (FST) constructed on the fractional derivative vector, instead of the gradient of the image.
ii. The spectrum aliasing problem when calculating the FST due to an inadequate sampling rate is prevented by doubling the sampling rate of FST (i.e., DFST) during the calculation of fractional derivatives according to the translation property of two-dimensional (2D) discrete Fourier transformation (2D-DFT).
iii. The performance of inferring the trend of diffusion is improved by utilizing a non-local filtering method to regularize the obtained DFST with the consideration that the visual elements or primitive texels of the textures are spatially repeated. Thus, after three modifications the resulting non-locally regularized DFST (i.e., NDFST) is obtained to effectively extract the subtle features of fine-grained textures.
The second contribution of this work is to plug the improved fractional tensor NDFST into an existing PDE to conduct an anisotropic diffusion to restore damaged textures. The most significant advantage is that such an improved tensor is tailored specifically to restore the damaged fine-grained textures and complex details in an image.
The remainder of this paper is organized as follows. We provide a brief review of the classic structure tensor in Section 2. In Section 3, we elaborate the details of three modifications of the improved NDFST and its numerical scheme. In Section 4, the obtained NDFST is plugged into a nonlinear PDE for fine-grained texture inpainting and its numerical implementation is presented. Generic numerical examples in Section 5 further highlight the remarkable inpainting qualities of the proposed PDE model. Conclusions are given in Section 6.
Section snippets
Brief review of classic structure tensor
The initial tensor product is computed from the gradient of , that is where , and is the transpose operator. Further, , where is 2D convolution, is a Gaussian kernel with standard deviation that depends on the noise scale in an initial image.
is a second-moment matrix of the image . Although contains no additional information than the gradient itself, it has the advantage that it can be smoothed without gradient
Improved structure tensor with three modifications
We present an improved NDFST, which incorporates three modifications of the classical ST to enable it to perform feature extraction to effectively process image textures, and which is particularly suitable for fine-grained texture images.
Inserting NDFST into the anisotropic PDE
In this section, inspired by existing PDE models, such as formula (2) and the literature [27], we plug the resulting NDFST into an anisotropic diffusion PDE for texture inpainting. is used to replace the traditional ST to extract texture features during diffusion. The obtained NDFST-based PDE model is
We can obtain the eigenvalues , (assuming ) and the corresponding mutually perpendicular eigenvectors , of the regularized tensor by means of
Experimental studies
We evaluated the effectiveness of the proposed method from two sides. One is to discuss the capability of feature extraction of the proposed NDFST, which plays a highly critical role in the diffusion process. Subsequently, the inpainting performance of the obtained NDFST-based PDE model for fine-grained texture images is concretely analyzed.
All computations were executed in MATLAB® R2014a on a 64-bit server with CPU-type Intel® Core (TM) i7-4790 @ 3.60 GHz with eight cores and 16 GB main memory.
Conclusions
In the traditional sense, the texture inpainting problem is more specifically suited to be addressed by exemplar-based methods; however, this kind of patch-wise method generally results in the defect of texture dislocation. Moreover, the pixel-wise PDE diffusion models are especially more effective at restoring cracked structures, but would introduce blurry deficiencies when repairing the fine-grained textures. In order to address these problems, we improved the traditional ST and proposed an
Acknowledgments
Funding: This work was partially supported by the Doctoral Scientific Research Foundation[no. 112-400211408] and the National Natural Science Foundation of China [nos. 61472319, 61502384 and 61501370].
Xiuhong Yang received her B.S. and M.S. degrees from the School of Information and Control Engineering, Xi’an University of Architecture and Technology and received her Ph.D. from the School of Electromechanical Engineering, Xidian University, China. She is an Assistant Professor with the School of Computer Science and Engineering, Xi’an University of Technology, China. Her research interests include digital image processing, multimedia communications, and digital image restoration. She is the
References (45)
- et al.
Nonlinear structure tensors
Image and Vision Comput.
(2006) - et al.
Nonlinear total variation based noise removal algorithms
Physica D
(1992) - et al.
Non-texture inpainting by curvature driven diffusions (CDD)
J. Visual Commun. Image Represent.
(2001) - et al.
Non-local adaptive structure tensors: Application to anisotropic diffusion and shock filtering
Image Vis. Comput.
(2011) - et al.
Fractional differentiation for edge detection
Signal Process.
(2003) A note on the gradient of a multi-image
Comput. Vision, Graphics, Image Process.
(1986)- N. Karianakis, P. Maragos, An integrated system for digital restoration of prehistoric theran wall paintings, in: Proc....
- et al.
Object removal by depth-wise image inpainting
Signal Image Video Process.
(2015) - S.-M. Muddala, R. Olsson, M. Sjöström, Disocclusion handling using depth-based inpainting, in: Proc. Int. Conf....
- et al.
Faithful disocclusion filling in depth image based rendering using superpixel-based inpainting
IEEE Trans. Multimedia
(2015)
Edge-directed error concealment
IEEE Trans. Circuits Syst. Video Technol.
Zoom-based super-resolution reconstruction approach using prior total variation
Opt. Eng.
Single image super-resolution via adaptive high-dimensional non-local total variation and adaptive geometric feature
IEEE Trans. Image Process.
Computer Vision
Textural features for image classification
IEEE Trans. Syst. Man, Cybern.
Fractional differential mask: a fractional differential-based approach for multiscale texture enhancement
IEEE Trans. Image Process.
Region filling and object removal by exemplar-based image inpainting
IEEE Trans. Image Process.
Image completion algorithm based on texture synthesis
J. Syst. Eng. Electron.
Improved exemplar based texture synthesis method for natural scene image completion
Int. J. Computer Sci.
Image completion by structure reconstruction and texture synthesis
Pattern Anal. Appl.
Cited by (0)
Xiuhong Yang received her B.S. and M.S. degrees from the School of Information and Control Engineering, Xi’an University of Architecture and Technology and received her Ph.D. from the School of Electromechanical Engineering, Xidian University, China. She is an Assistant Professor with the School of Computer Science and Engineering, Xi’an University of Technology, China. Her research interests include digital image processing, multimedia communications, and digital image restoration. She is the corresponding author of this paper.
Baolong Guo received his B.S., M.S., and Ph.D. degrees from the School of Electromechanical Engineering, Xidian University. He is a Professor and Ph.D. supervisor with the school of Aerospace Science and Technology, Xidian University, China. His research interests include intelligent control, signal analysis and processing, and image engineering.
Zhaolin Xiao received Bachelor’s and Master’s degrees from the Department of Computer Science and Engineering, Xi’an Technological University in 2006 and 2009, respectively. He received his Ph.D. degree from the Department of Computer Science and Engineering, Northwestern Polytechnical University in 2014. He then joined Xi’an University of Technology as a Lecturer. He is a Member of the China Computer Federation (CCF). He worked as a visiting scholar in the School of Computer Science, The University of Wisconsin-Madison, USA, in 2011 and 2012. In 2012, he visited the Department of Computer and Information Sciences, University of Delaware, for one month. Dr. Xiao’s research interests include image processing, computer vision, and computational photography. At present, his research focus is on image-based 3D reconstruction, light field imaging, and processing.
Wei Liang is a lecturer in the Virtual Reality and Visualization Institute, Xi’an University of Technology. She received her B.S. degree in computer science and engineering from Huaihai Institute of Technology in 2008 and her Ph.D. degree in computer architecture from Xidian University in 2014. Her current research interests include multispectral image compression, multispectral image registration, and color reproduction.
- 1
Fractional structure tensor.
- 2
Double-sampling fractional structure tensor.
- 3
Nonlocally-regularized double-sampling fractional structure tensor.