A new image segmentation algorithm with applications to image inpainting
Introduction
During the past decades, image segmentation and edge detection have been two important and challenging topics. The main idea is to produce a partition of an image such that each category or region is homogeneous with respect to some measures. The processed image can be useful for posterior image processing treatments. Many techniques have been studied in this field, from different perspectives, and several different disciplines have been involved. For example, unsupervised clustering techniques (Jain et al., 1999), and morphological multifractal estimation for image segmentation (Xia et al., 2006) have been used, among others.
Two-dimensional autoregressive (AR-2D) models are one way to represent the image intensity of a given picture by a small number of parameters. This class of models, pioneered by Whittle (1954), has been studied in several different disciplines. Kashyap and Eom (1988) developed an image restoration algorithm based on robust estimation of a two-dimensional autoregressive model. Cariou and Chehdi (2008) used a two-dimensional autoregressive model to perform unsupervised texture segmentation. Allende et al. (2001) generalized the algorithm proposed by Kashyap and Eom (1988), using the generalized M estimators to deal with the effect caused by additive contamination. Later on, Ojeda et al. (2002) developed robust autocovariance (RA) estimators for AR-2D processes. Several theoretical contributions have been suggested in the literature. For example, Baran et al. (2004) investigated asymptotic properties of a nearly unstable sequence of stationary spatial autoregressive processes. Other contributions and applications of spatial autoregressive moving average (ARMA) processes can be found in Tjostheim (1978), Guyon (1982), Basu and Reinsel (1993), Martin (1996), Francos and Friedlander (1998), Illig and Truong-Van (2006).
In this paper we focus our attention on segmentation and edge detection of texture images. A new image segmentation algorithm that highlights the edges of an image is described. The algorithm consists in locally fitting a two-dimensional autoregressive model to the original image. That is, the original image is divided into small regions and an AR-2D model is fitted to each of these regions. A new image is generated, putting together all images generated by fitting the local AR-2D models to the original one. Then the autoregressive residual image is computed. As a result, the original borders are highlighted and the areas with different textures are noticed. One advantage of our algorithm is its simplicity, since in practice a large class of images can be well represented by AR-2D models with less than four parameters. The fitted AR-2D model plays an important role in the segmentation process because the quality of the fitted model significantly affects the segmentation results.
Numerical studies with real images are offered to inspect the advantages and limitations of our proposal. Several images were processed to gain a better insight into the performance of the algorithm. In each case, AR-2D models were used to represent the original patterns considering least squares estimation for the parameters. The same algorithm is studied when the image is blurred by additive contamination. In this case it is well known that traditional methods of estimation of AR-2D models yield estimators that are highly sensitive to outliers. The effect on the segmentation produced by the proposed algorithm is discussed. We also present experiments in which robust estimation has been used instead of traditional least squares (LS) and maximum likelihood (ML) estimations.
Inpainting is a technique to reconstruct damaged or missed portions of an image. A novel application using real images is shown in this framework. Our algorithm is able to detect some patterns on images that have been previously processed by inpainting techniques to fill certain small image gaps, highlighting the gaps or imperfections existing in the original image.
The paper is organized as follows. In Section 2, we give an overview of the spatial AR processes. In Section 3, the main robust estimation methods are reviewed. Section 4 describes the new image segmentation algorithm. In Section 5, a simulation experiment is developed to illustrate the segmentation procedure. In Section 6, the performance of our algorithm is inspected under additive contamination. Section 7 presents an application of our algorithm for image inpainting. We conclude and present possible extensions of this work in Section 8.
Section snippets
The spatial ARMA processes
In order to represent images using models that are statistically treatable, three classes of model have been proposed. Whittle (1954) studied simultaneous autoregressive (AR) models; Besag (1974) introduced conditional autoregressive models. Moving average (MA) models were studied by Haining (1978).
Spatial autoregressive moving average (ARMA) processes have also been studied in the context of random fields indexed over , where is endowed with the usual partial order; that is; for
Robust parametric estimation
Innovation outliers (IOs) and additive outliers (AOs) are well known in time series (Fox, 1972). The same notion of data contamination has been studied for spatial processes. A more recent discussion about types of contamination in time series can be found in Chang et al. (1988) and Chen and Lui (1993). The definitions of outliers have been extended to a multivariate framework and the effects of multivariate outliers on the joint and marginal models have been examined by Tsay et al. (2000).
It
The new algorithm
In this section we present two algorithms. The first one produces a local approximation of images by using unilateral AR-2D processes. The second one is the new segmentation algorithm.
The first algorithm is based on the fact that it is possible to represent any image by using unilateral AR-2D processes. This image is called a local AR-2D approximated image by using blocks.
Let be the original image, and let where, for all ,
Numerical experiments
In this section, we present numerical experiments that illustrate the performance of Algorithm 2. The images considered in the experiment were taken from the USC-SIPI image database http://sipi.usc.edu/database/. Fig. 1(a) shows an original image of size 512×512. Fig. 1(b) shows the image produced by Algorithm 1 using a moving window of size 7×7. In Fig. 1(c), we display the residual autoregressive image yielded by Algorithm 2. Notice from Fig. 1(b) that the local approximated image is visually
A segmentation algorithm for contaminated images
The analysis of contaminated images is of great interest in several areas of research. For example, the reconstruction of contaminated images is relevant in image modeling (Allende and Galbiati, 2004, Vallejos and Mardesic, 2004), and in general the reduction of the noise produced by interferences taking place in the processes of obtaining the physical image and of transmitting it electronically plays an important role in the literature (Bustos, 1997).
In this section, using real images, we
An application to image inpainting
Inpainting, the technique of reconstructing small damaged portions of an image, has received considerable attention in recent years. This consists of filling in the missing areas or modifying the damaged ones in a non-detectable way for an observer not familiar with the original images. Inpainting serves a wide range of applications, such as restoration of photographs, films and paintings, and removal of occlusions, such as text, subtitles, stamps and publicity, from images. In addition,
Final comments
In this paper, we have introduced a new algorithm to perform image segmentation. The foundations of this algorithm are random field theory and robustness for spatial autoregressive processes. In the light of the examples shown in Sections 5 Numerical experiments, 6 A segmentation algorithm for contaminated images, we conclude that our algorithm is able to highlight the borders and contours of a large class of images. The LS estimation looks to be appropriate in this framework. On the other
Acknowledgements
We thank an associate editor and the referees for helpful comments and suggestions. The first author was supported by Secyt-UNC grant (05/B412), Argentina. The second author was partially supported by Fondecyt grant 11075095, Chile.
References (41)
- et al.
Robust image modeling on image processing
Pattern Recognition Letters
(2001) - et al.
A non-parametric filter for digital image restoration, using cluster analysis
Pattern Recognition Letters
(2004) - et al.
Asymptotic inference for a nearly unstable sequence of stationary spatial AR models
Statistics & Probability Letters
(2004) - et al.
Asymptotic behavior of RA-estimates in autoregressive processes
Journal of Statistical Planning and Inference
(2009) - et al.
Unsupervised texture segmentation/classification using 2-D autoregressive modeling and the stochastic expectation–maximization algorithm
Pattern Recognition Letters
(2008) Some results on unilateral ARMA lattice processes
Journal of Statistical Planning and Inference
(1996)- Ballester, C., Caselles, V., Verdera, J., Bertalmio, M., Sapiro, G., 2001. A variational model for filling-in gray...
- et al.
Properties of the spatial unilateral first-order ARMA model
Advances in Applied Probability
(1993) - Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C., 2000. Image inpainting. In: Proc. ACM Conf. Comp. Graphics,...
- Bertalmio, M., Sapiro, G., Vese, O., 2003. Simultaneous structure and texture image inpainting. In: Proc. Conf. Comp...
Spatial interaction and the statistical analysis of lattice systems (with discussion)
Journal of the Royal Statistical Society Series B
Robust estimates for ARMA models
Journal of the American Statistical Association
Robust Statistics in SAR image processing
ESA-SP
Spatial ARMA models and its applications to image filtering
Brazilian Journal of Probability and Statistics
Estimation of time series parameters in the presence of outliers
Technometrics
Joint estimation of model parameters and outliers in time series
Journal of the American Statistical Association
On the asymptotic distribution of mean, autocovariance, autocorrelation, crosscovariance and impulse response estimators of a stationary multidimensional random field
Communications in Statistics. Theory and Methods
Region Filling and object removal by exemplar-based image inpainting
IEEE Transactions on Image Processing
Spatial analysis of field experiments—an extension to two dimensions
Biometrics
Parameter estimation of two-dimensional moving average random fields
IEEE Transaction on Signal Processing
Cited by (22)
2-D Rayleigh autoregressive moving average model for SAR image modeling
2022, Computational Statistics and Data AnalysisCitation Excerpt :The parametric representation of two-dimensional homogeneous random fields considering two-dimensional (2-D) autoregressive moving average (ARMA) models is frequently adopted for image processing (Bustos et al., 2009a; Kizilkaya and Kayran, 2005; Ojeda et al., 2010; Bustos et al., 2009b), including (i) modeling (Bustos et al., 2009a; Rosenfeld, 2014); (ii) compression (Nijim et al., 1996); (iii) encoding (Chung and Kanefsky, 1992); and (iv) restoration (Lim, 1990; Vallejos and Mardesic, 2004).
Asymptotic properties of BMM-estimator in bidimensional autoregressive processes
2020, Journal of Statistical Planning and InferenceCitation Excerpt :In the area of image processing and computer vision, the use and development of techniques called “robust” are frequent, not referring to the term robustness imperiously from a formal statistical perspective. Within this framework, the classic tools do not consider the structure or topology inherent to the images and, in most situations, the models require strong hypotheses about the laws that govern the observed process (Alata and Olivier, 2003; Bustos et al., 2009a; Dormann et al., 2007; Latha et al., 2014; Ojeda et al., 2010; Quintana et al., 2011; Sahu et al., 2015; Sain and Cressie, 2007; Vallejos and Mardesic, 2004) and (Zielinski et al., 2010). From a formal perspective, in the context of robust estimators of the AR-2D model parameters with a finite and arbitrary number of parameters, at least three estimators have been defined and studied: the M, GM and, RA estimators.
A spatial concordance correlation coefficient with an application to image analysis
2020, Spatial StatisticsLarge gap imputation in remote sensed imagery of the environment
2012, Computational Statistics and Data AnalysisCitation Excerpt :In that case, imputation is made by pixel value prediction, estimating the parameters of the model from the regions of good data. In the case of small gaps, spatial models for inpainting optical images have been proposed by Ojeda et al. (2010) and Bustos et al. (2009), but its extensions to Landsat data problems are still in the developing stage. In relation to Method C, we acknowledge that some accurate previous knowledge about the map of classes of the nondamaged part of the image is mandatory.
Context-based image segment labeling (CBISL)
2020, arXiv