Film line scratch detection using texture and shape information

Abstract

A scratch detection and restoration is very important, as scratches are the most common form of degradation of old films. The goal of the current study is to develop a fully automated system that can detect all types of scratch with a low computational cost. This is achieved by defining the texture and shape properties from spatial domain, then using these for scratch detection. The proposed method involves two procedures: (1) the input image is divided into scratches and non-scratches using a neural network (NN)-based texture classifier and (2) some false alarms are removed by shape filtering using a morphological filter with new structuring elements defined based on the shape characteristics of scratches. To assess the validity of the proposed method, experiments were performed with several films, and the results showed that its performance was superior to that of other method.

Introduction

Film restoration involves detecting the location and extent of defected regions in a movie, and then reconstructing the lost information in each region. In recent years, film restoration has attracted increasing research attention in order to provide high quality multimedia services (Kokaram, 1998, Tegolo and Isgro, 2001, D’amore et al., 2007, Kemal Gullu, 2008, Schallauer et al., 2007, Vittoria Bruni et al., 2004, Ren and Vlanchos, 2007, Maddalena, 2001).

Old films are usually degraded by dust, scratches, and flicks, among which scratches are the most common form of degradation (Kokaram, 1998, Juyeux, 2001, Schallauer et al., 2007, Kao et al., 2007). Thus, an automatic scratch detection and restoration system is needed, and then the detection is the focus of the current paper.

Scratches are usually generated by mechanical rubbing while copying a film, and appear in the direction of the film strip in successive frames. Fig. 1 shows some examples of scratches.

As shown in Fig. 1, scratches are easily visible as vertical lines of bright or dark intensity, oriented vertically over much of the image. Therefore, the representative characteristics of a scratch can be defined as follows: (1) it has a lower or higher brightness than the neighboring pixels, (2) usually appears as a vertically long thin line, and (3) has a temporal continuity, i.e. it appears in successive frames.

Among these characteristics, (1) and (2) are textural and morphological in the spatial domain, while (3) is a continuous characteristic in the temporal domain. Consequently, these characteristics can help reduce the complexity of detection and facilitate discrimination between scratched and non-scratched regions.

Various scratch detection systems have already been developed based on these characteristics (Kokaram, 1998, Tegolo and Isgro, 2001, Lee, 2006, Juyeux, 2001, Schallauer et al., 2007, Kao et al., 2007) and are generally composed of a candidate detection step and verification step. Candidate scratch regions are identified by finding the local extreme of a frame along the x-axis or spatial discontinuities, which are obtained using median filtering, ridge detection, and extreme gray scale morphology (Kokaram, 1996, Jain, 1989, Juyeux, 2001, Tekalp, 1995). While this first step is relative simple and effective for finding scratches, it also includes many false alarms due to lines that occur as a natural part of a scene. As a solution to this problem, a neural network (NN)-based texture classifier was used to detect scratches in the previous work (Sin Kuk Kang, 2004). Although it can reduce many false alarms, it requires much computational cost to scan a whole image using NN and still includes some false alarms.

Therefore, a verification step is needed to distinguish real scratches from natural scene components. One approach to remove false alarms is to use the temporal characteristic that scratches persist nearby or in the same location across several frames. In (Juyeux, 2001, Kao et al., 2007), various effective methods for tracking detected scratches are presented using Kalman filtering and block matching algorithms, yet these methods involve a high computational cost. Moreover, motion estimation remains a problem in ongoing research. An alternative approach for detecting real scratches is to use the particular shape properties of scratches, such as their width, angle to the vertical, and height (Kokaram, 1996, Kokaram, 1998, Vittoria Bruni et al., 2004, Sin Kuk Kang, 2004). The height and width of scratches can be determined using empirical examination, or Weber’s law and a Bayesian refinement strategy can be employed to find thresholds for tuning the shapes. Nonetheless, while effective in rejecting some false alarms, these spatial methods require the manual tuning of various shape thresholds. Furthermore, both approaches can only find positions of columns including scratches, rather than accurate positions of rows and columns, thereby creating an additional computational cost in the restoration stage.

Accordingly, this paper presents a new scratch detection method that can detect the accurate position of all kinds of scratches in frames from old films. As the main goal is the accurate detection of all scratches without a high computational cost or human intervention, this is achieved by defining the texture and shape properties from the spatial domain, then using these for scratch detection. Here, the textural property of scratches is identified through machine learning and used to detect scratch candidates, instead of simple colors, and the shape property of scratches is described by newly defined structuring elements and used to detect false alarms. While the first property reduces the number of false dismissals, the second enables the correct discrimination of real scratches from false alarms without the use of temporal information or human intervention. Thus, the proposed system is composed of two major modules: a neural network-based texture classifier and morphology-based shape filter with multiple structuring elements. The texture classification step divides the input image into scratched regions and non-scratched regions using the texture property of scratches. Then, the classifier is just applied to edge pixels to reduce the computational cost. Meanwhile, the shape filtering step confirms the classified scratched regions using a morphological filter with new structuring elements designed based on the shape characteristics of scratches.

To assess the effectiveness of the proposed method, experiments were performed using several old films and a synthetic film, and the results compared with those from Kokaram’s method. The comparison revealed that the proposed method was superior to the conventional method. In addition, it was demonstrated that the proposed method could be applied to commercial systems, as it provided accurate detection without human intervention.

The remainder of this paper is organized as follows. Section 2 provides an overview of the proposed system, and the texture classification and shape filtering steps are then described in Sections 3 Overview of proposed system texture classification, 4 Shape filtering, respectively. Experimental results are presented in Section 5, and some final conclusions and areas for future work given in Section 6.