A new image segmentation algorithm with applications to image inpainting

https://doi.org/10.1016/j.csda.2010.03.021Get rights and content

Abstract

This article describes a new approach to perform image segmentation. First an image is locally modeled using a spatial autoregressive model for the image intensity. Then the residual autoregressive image is computed. This resulting image possesses interesting texture features. The borders and edges are highlighted, suggesting that our algorithm can be used for border detection. Experimental results with real images are provided to verify how the algorithm works in practice. A robust version of our algorithm is also discussed, to be used when the original image is contaminated with additive outliers. A novel application in the context of image inpainting is also offered.

Introduction

During the past decades, image segmentation and edge detection have been two important and challenging topics. The main idea is to produce a partition of an image such that each category or region is homogeneous with respect to some measures. The processed image can be useful for posterior image processing treatments. Many techniques have been studied in this field, from different perspectives, and several different disciplines have been involved. For example, unsupervised clustering techniques (Jain et al., 1999), and morphological multifractal estimation for image segmentation (Xia et al., 2006) have been used, among others.

Two-dimensional autoregressive (AR-2D) models are one way to represent the image intensity of a given picture by a small number of parameters. This class of models, pioneered by Whittle (1954), has been studied in several different disciplines. Kashyap and Eom (1988) developed an image restoration algorithm based on robust estimation of a two-dimensional autoregressive model. Cariou and Chehdi (2008) used a two-dimensional autoregressive model to perform unsupervised texture segmentation. Allende et al. (2001) generalized the algorithm proposed by Kashyap and Eom (1988), using the generalized M estimators to deal with the effect caused by additive contamination. Later on, Ojeda et al. (2002) developed robust autocovariance (RA) estimators for AR-2D processes. Several theoretical contributions have been suggested in the literature. For example, Baran et al. (2004) investigated asymptotic properties of a nearly unstable sequence of stationary spatial autoregressive processes. Other contributions and applications of spatial autoregressive moving average (ARMA) processes can be found in Tjostheim (1978), Guyon (1982), Basu and Reinsel (1993), Martin (1996), Francos and Friedlander (1998), Illig and Truong-Van (2006).

In this paper we focus our attention on segmentation and edge detection of texture images. A new image segmentation algorithm that highlights the edges of an image is described. The algorithm consists in locally fitting a two-dimensional autoregressive model to the original image. That is, the original image is divided into small regions and an AR-2D model is fitted to each of these regions. A new image is generated, putting together all images generated by fitting the local AR-2D models to the original one. Then the autoregressive residual image is computed. As a result, the original borders are highlighted and the areas with different textures are noticed. One advantage of our algorithm is its simplicity, since in practice a large class of images can be well represented by AR-2D models with less than four parameters. The fitted AR-2D model plays an important role in the segmentation process because the quality of the fitted model significantly affects the segmentation results.

Numerical studies with real images are offered to inspect the advantages and limitations of our proposal. Several images were processed to gain a better insight into the performance of the algorithm. In each case, AR-2D models were used to represent the original patterns considering least squares estimation for the parameters. The same algorithm is studied when the image is blurred by additive contamination. In this case it is well known that traditional methods of estimation of AR-2D models yield estimators that are highly sensitive to outliers. The effect on the segmentation produced by the proposed algorithm is discussed. We also present experiments in which robust estimation has been used instead of traditional least squares (LS) and maximum likelihood (ML) estimations.

Inpainting is a technique to reconstruct damaged or missed portions of an image. A novel application using real images is shown in this framework. Our algorithm is able to detect some patterns on images that have been previously processed by inpainting techniques to fill certain small image gaps, highlighting the gaps or imperfections existing in the original image.

The paper is organized as follows. In Section 2, we give an overview of the spatial AR processes. In Section 3, the main robust estimation methods are reviewed. Section 4 describes the new image segmentation algorithm. In Section 5, a simulation experiment is developed to illustrate the segmentation procedure. In Section 6, the performance of our algorithm is inspected under additive contamination. Section 7 presents an application of our algorithm for image inpainting. We conclude and present possible extensions of this work in Section 8.

Section snippets

The spatial ARMA processes

In order to represent images using models that are statistically treatable, three classes of model have been proposed. Whittle (1954) studied simultaneous autoregressive (AR) models; Besag (1974) introduced conditional autoregressive models. Moving average (MA) models were studied by Haining (1978).

Spatial autoregressive moving average (ARMA) processes have also been studied in the context of random fields indexed over Zd,d≥2, where Zd is endowed with the usual partial order; that is; for s=(s1,

Robust parametric estimation

Innovation outliers (IOs) and additive outliers (AOs) are well known in time series (Fox, 1972). The same notion of data contamination has been studied for spatial processes. A more recent discussion about types of contamination in time series can be found in Chang et al. (1988) and Chen and Lui (1993). The definitions of outliers have been extended to a multivariate framework and the effects of multivariate outliers on the joint and marginal models have been examined by Tsay et al. (2000).

It

The new algorithm

In this section we present two algorithms. The first one produces a local approximation of images by using unilateral AR-2D processes. The second one is the new segmentation algorithm.

The first algorithm is based on the fact that it is possible to represent any image by using unilateral AR-2D processes. This image is called a local AR-2D approximated image by using blocks.

Let Z=[Zm,n]0≤m≤M−1,0≤n≤N−1, be the original image, and let X=[Xm,n]0≤m≤M−1,0≤n≤N−1, where, for all 0≤m≤M−1,0≤n≤N−1, Xm,n=Zm,

Numerical experiments

In this section, we present numerical experiments that illustrate the performance of Algorithm 2. The images considered in the experiment were taken from the USC-SIPI image database http://sipi.usc.edu/database/. Fig. 1(a) shows an original image of size 512×512. Fig. 1(b) shows the image produced by Algorithm 1 using a moving window of size 7×7. In Fig. 1(c), we display the residual autoregressive image yielded by Algorithm 2. Notice from Fig. 1(b) that the local approximated image is visually

A segmentation algorithm for contaminated images

The analysis of contaminated images is of great interest in several areas of research. For example, the reconstruction of contaminated images is relevant in image modeling (Allende and Galbiati, 2004, Vallejos and Mardesic, 2004), and in general the reduction of the noise produced by interferences taking place in the processes of obtaining the physical image and of transmitting it electronically plays an important role in the literature (Bustos, 1997).

In this section, using real images, we

An application to image inpainting

Inpainting, the technique of reconstructing small damaged portions of an image, has received considerable attention in recent years. This consists of filling in the missing areas or modifying the damaged ones in a non-detectable way for an observer not familiar with the original images. Inpainting serves a wide range of applications, such as restoration of photographs, films and paintings, and removal of occlusions, such as text, subtitles, stamps and publicity, from images. In addition,

Final comments

In this paper, we have introduced a new algorithm to perform image segmentation. The foundations of this algorithm are random field theory and robustness for spatial autoregressive processes. In the light of the examples shown in Sections 5 Numerical experiments, 6 A segmentation algorithm for contaminated images, we conclude that our algorithm is able to highlight the borders and contours of a large class of images. The LS estimation looks to be appropriate in this framework. On the other

Acknowledgements

We thank an associate editor and the referees for helpful comments and suggestions. The first author was supported by Secyt-UNC grant (05/B412), Argentina. The second author was partially supported by Fondecyt grant 11075095, Chile.

References (41)

  • J. Besag

    Spatial interaction and the statistical analysis of lattice systems (with discussion)

    Journal of the Royal Statistical Society Series B

    (1974)
  • O. Bustos et al.

    Robust estimates for ARMA models

    Journal of the American Statistical Association

    (1986)
  • O. Bustos

    Robust Statistics in SAR image processing

    ESA-SP

    (1997)
  • O. Bustos et al.

    Spatial ARMA models and its applications to image filtering

    Brazilian Journal of Probability and Statistics

    (2009)
  • I. Chang et al.

    Estimation of time series parameters in the presence of outliers

    Technometrics

    (1988)
  • C. Chen et al.

    Joint estimation of model parameters and outliers in time series

    Journal of the American Statistical Association

    (1993)
  • B. Choi

    On the asymptotic distribution of mean, autocovariance, autocorrelation, crosscovariance and impulse response estimators of a stationary multidimensional random field

    Communications in Statistics. Theory and Methods

    (2000)
  • A. Criminisi et al.

    Region Filling and object removal by exemplar-based image inpainting

    IEEE Transactions on Image Processing

    (2004)
  • B.R. Cullis et al.

    Spatial analysis of field experiments—an extension to two dimensions

    Biometrics

    (1991)
  • J. Francos et al.

    Parameter estimation of two-dimensional moving average random fields

    IEEE Transaction on Signal Processing

    (1998)

    • Cited by (22)

    • 2-D Rayleigh autoregressive moving average model for SAR image modeling

      2022, Computational Statistics and Data Analysis
      Citation Excerpt :

      The parametric representation of two-dimensional homogeneous random fields considering two-dimensional (2-D) autoregressive moving average (ARMA) models is frequently adopted for image processing (Bustos et al., 2009a; Kizilkaya and Kayran, 2005; Ojeda et al., 2010; Bustos et al., 2009b), including (i) modeling (Bustos et al., 2009a; Rosenfeld, 2014); (ii) compression (Nijim et al., 1996); (iii) encoding (Chung and Kanefsky, 1992); and (iv) restoration (Lim, 1990; Vallejos and Mardesic, 2004).

    • Asymptotic properties of BMM-estimator in bidimensional autoregressive processes

      2020, Journal of Statistical Planning and Inference
      Citation Excerpt :

      In the area of image processing and computer vision, the use and development of techniques called “robust” are frequent, not referring to the term robustness imperiously from a formal statistical perspective. Within this framework, the classic tools do not consider the structure or topology inherent to the images and, in most situations, the models require strong hypotheses about the laws that govern the observed process (Alata and Olivier, 2003; Bustos et al., 2009a; Dormann et al., 2007; Latha et al., 2014; Ojeda et al., 2010; Quintana et al., 2011; Sahu et al., 2015; Sain and Cressie, 2007; Vallejos and Mardesic, 2004) and (Zielinski et al., 2010). From a formal perspective, in the context of robust estimators of the AR-2D model parameters with a finite and arbitrary number of parameters, at least three estimators have been defined and studied: the M, GM and, RA estimators.

    • A spatial concordance correlation coefficient with an application to image analysis

      2020, Spatial Statistics

    • Large gap imputation in remote sensed imagery of the environment

      2012, Computational Statistics and Data Analysis
      Citation Excerpt :

      In that case, imputation is made by pixel value prediction, estimating the parameters of the model from the regions of good data. In the case of small gaps, spatial models for inpainting optical images have been proposed by Ojeda et al. (2010) and Bustos et al. (2009), but its extensions to Landsat data problems are still in the developing stage. In relation to Method C, we acknowledge that some accurate previous knowledge about the map of classes of the nondamaged part of the image is mandatory.

    • 2-D Rayleigh Autoregressive Moving Average Model for SAR Image Modeling

      2022, arXiv

    • Context-based image segment labeling (CBISL)

      2020, arXiv
    View all citing articles on Scopus
    View full text
    Copyright © 2010 Elsevier B.V. All rights reserved.