Principles of Constructing a Performance Evaluation Protocol for Graphics Recognition Algorithms

Abstract

Graphics recognition is a process that takes as input a raster level image consisting of pixels or a vector level drawing consisting of symbolic primitives. It groups primitives on the input into higher order graphic entities, and identifies them with known objects on the basis of the matching basic features while allowing for parameter variation. The graphic objects that may be recognized from the input include text (character) regions, lines of various shapes (e.g., circular arcs and polylines) and styles (e.g., dashed lines and dash-dotted lines), special symbols, dimension sets, etc. Graphics recognition techniques have been developed for many years. However, the performance of most of these techniques are at best known only from the reports of their developers, based on their own perceptual, subjective, and qualitative human vision evaluation. Objective evaluations and quantitative comparisons among them are not available. This is due to the lack of protocols that provide for quantitative measurements of their interesting metrics, a sound methodology for acquiring appropriate ground truth data, and adequate methods for matching the ground truths with the recognized graphic objects. To further advance the research on graphics recognition, to fully comprehend and reliably compare the performance of graphics recognition algorithms, and to help select, improve, and even design new algorithms to be applied in new systems designed for some specific application, the establishment of objective and comprehensive evaluation protocols and a resulting performance evaluation methodology are strongly required. Groups that have reported research on performance evaluation of graphics recognition algorithms include (Kong et al., 1996; Hori and Doermann, 1996; Liu and Dori, 1997; Liu and Dori, 1998a) and (Philips et al., 1998). However, their researches are only on performance evaluation of recognition algorithms for some specific classes of graphic objects. The protocols of (Kong et al., 1996; Hori and Doermann, 1996; Liu and Dori, 1997) are aimed at performance evaluation of line detection algorithms. Liu and Dori (1998a) propose a protocol for text segmentation evaluation. Philips et al. (1998) propose a performance evaluation protocol for engineering drawings recognition systems, which includes performance evaluation of both line detection and text segmentation capabilities. There is no common methodology that abstracts the performance genericity of graphics recognition algorithms and can be generally applied to their performance evaluation. Based on the observed genericity of graphics recognition (Ablameyko, 1996; Liu and Dori, 1998b) we propose a methodology for performance evaluation of graphics recognition algorithms. The methodology materializes an objective-driven evaluation philosophy that is based on definitions of a matching degree and comprehensive performance metrics.