The comparison index: A tool for assessing the accuracy of image segmentation

https://doi.org/10.1016/j.jag.2006.10.002Get rights and content

Abstract

Segmentation algorithms applied to remote sensing data provide valuable information about the size, distribution and context of landscape objects at a range of scales. However, there is a need for well-defined and robust validation tools to assessing the reliability of segmentation results. Such tools are required to assess whether image segments are based on ‘real’ objects, such as field boundaries, or on artefacts of the image segmentation algorithm. These tools can be used to improve the reliability of any land-use/land-cover classifications or landscape analyses that is based on the image segments.

The validation algorithm developed in this paper aims to: (a) localize and quantify segmentation inaccuracies; and (b) allow the assessment of segmentation results on the whole. The first aim is achieved using object metrics that enable the quantification of topological and geometric object differences. The second aim is achieved by combining these object metrics into a ‘Comparison Index’, which allows a relative comparison of different segmentation results. The approach demonstrates how the Comparison Index CI can be used to guide trial-and-error techniques, enabling the identification of a segmentation scale H that is close to optimal. Once this scale has been identified a more detailed examination of the CI–H- diagrams can be used to identify precisely what H value and associated parameter settings will yield the most accurate image segmentation results.

The procedure is applied to segmented Landsat scenes in an agricultural area in Saxony-Anhalt, Germany. The segmentations were generated using the ‘Fractal Net Evolution Approach’, which is implemented in the eCognition software.

Introduction

Image segmentation algorithms such as those contained within eCognition™ are increasingly popular for a wide range of image processing tasks, and the advantages of working with image segments, rather than individual pixels are widely recognized (Fortin et al., 2000, Shi et al., 2005). However, there is a wide range of variables to manipulate, whereas segmenting an image and identifying an ‘optimal’ result can be difficult. The tools developed in this paper aim to make this process more objective and rigorous.

In this study, image segmentation algorithms were applied to Landsat TM data for an agricultural region in Saxony-Anhalt, Germany. The accuracy of the field boundary delineation was highly important because of its impact on the accuracy of the sediment/pollutant transport model that uses the field boundaries as model input (Van Oost et al., 2000, Takken et al., 2001). In addition, measures of soil protection legislation like ‘European Common Agricultural Policy’ or ‘Good Farming Practice’ (EU, 2002) refer to up-to-date field shapes. However, the existing public vector data sets provided by land surveys are not able to define field boundaries accurately. This is mainly due to the large temporal variations of crop structures (Mysiak et al., 2004). Satellite sensor data have the capacity to fill this ‘temporal gap’ thereby providing up-to-date field boundary information.

To assess the accuracy of the field boundary delineation we focus on the geometric quality of a single class (cp. Zhan et al., 2005) rather than classification accuracy (sensu Foody, 2002). The image segmentation was carried out using the ‘fractal net evolution approach’ (FNEA) (Benz et al., 2004). FNEA is constrained using a set of user-defined parameter settings, which affect the segmentation results. In this paper, we investigate the influence of segmentation parameter settings on segmentation results with a view to identifying the parameter settings that provide optimal segmentation results for a specific target. The main objectives of this study are:

  • (1)

    the development of a user-friendly evaluation procedure to visualize and quantify segmentation inaccuracies. Inaccuracies refer to over-segmentations or under-segmentations, which stand for generating too many or too few segments (Delves et al., 1992). This level is referred to as local validation, because single objects are considered;

  • (2)

    the assessment of segmentation results on the whole, by which we want to achieve an optimal parameter setting of the applied segmentation method. This level is referred to as global validation, because the entire image is considered.

These objectives were achieved by visualizing two object metrics (local validation) and using the Site Comparison Method SICOM (Deumlich et al., 2006) which aggregates the classified object metrics to a map complexity metric Comparison Index CI (global validation). The extrapolation of the complexity metric to the whole study area was realized using random sampling methodology (Congalton and Green, 1990, Stehmann, 1992).

Section snippets

Image segmentation

There are various methods for automatic field detection that are based on the application of segmentation algorithms to remote sensing data (e.g., Fuller et al., 2002, Evans et al., 2002, Betenuth, 2004, Mueller et al., 2004, Devereux et al., 2004). Image segmentation is a spatial clustering technique, which leads to a complete image sub-division into non-overlapping regions or segments. The wide variety of segmentation approaches can be distinguished in two broad categories (Fortin et al., 2000

Study area

The study area (435 km2) is situated in the south of the German state Saxony-Anhalt near the city of Halle (Saale) (Fig. 3). The accurate delineation of agricultural field boundaries is important because this area is a study site of a major soil erosion study within a project about ‘Integrated River Basin Management on the example of the Saale River Basin’ (Rode et al., 2002). The field boundaries are used here as model inputs into empirical and physics-based erosion modelling (cp. Merritt et

Reference objects and segmentation

The results of manual field detection are visualized in Fig. 4a. Reference objects (400) were digitized. The validation procedure will be exemplified by the reference object shown in red in the south of the study area (Fig. 4b). The round field boundaries are a function of the circular buffer of 1000 m that we used around the randomly selected points.

The segmentation parameter settings are listed in Table 1. The H-, wcompt-and wshape-parameters were generated using trial-and-error tests to

Conclusions

The main objective of this paper was the determination of an optimal parameter setting of the FNEA-segmentation method in eCogntion™ based on the assumption that an optimal parameter setting is reached when over-and under-segmentation are balanced. The approach developed in this paper demonstrates how a two-stage process can be used to identify a segmentation scale H that is close to optimal using trial-and-error tests in combination with the CI metric. Once this scale has been identified a

References (37)

  • I. Takken et al.

    Effects of tillage on runoff and erosion patterns

    Soil Tillage Res.

    (2001)
  • M. Baatz et al.
  • Betenuth, M., 2004. Extraction of field boundaries and wind erosion obstacles from aerial imagery, in: Seyfert, E.,...
  • C. Bishop

    Neural Networks for Pattern Recognition

    (1995)
  • R. Congalton et al.

    Assessing The Accuracy of Remotely Sensed Data: Principles and Practices

    (1990)
  • L.M. Delves et al.

    Comparing the performance of SAR image segmentation algorithms

    Int. J. Remote Sens.

    (1992)
  • EU, 2002. European communities—the Water Framework Directive (WFD) and tools within the Common Agricultural Policy...
  • C. Evans et al.

    Segmenting multispectral Landsat TM images into field units

    IEEE Trans. Geosci. Remote Sens.

    (2002)
  • Cited by (203)

    • Object detection in hyperspectral images

      2022, Digital Signal Processing: A Review Journal
    • Detecting and mapping karst landforms using object-based image analysis: Case study: Takht-Soleiman and Parava Mountains, Iran

      2022, Egyptian Journal of Remote Sensing and Space Science
      Citation Excerpt :

      In the third stage, the FSE method is applied to determine the accuracy of the classification results (Fig. 5). Image segmentation is one of the most critical steps in object-based image analysis (Möller et al., 2007). The main aim of this process is to subdivide the digital image into smaller objects based on the spatial and spectral information (Feizizadeh et al., 2021b).

    • Automated delineation of agricultural field boundaries from Sentinel-2 images using recurrent residual U-Net

      2021, International Journal of Applied Earth Observation and Geoinformation
    View all citing articles on Scopus
    View full text