Elsevier

Pattern Recognition Letters

Volume 52, 15 January 2015, Pages 72-79
Pattern Recognition Letters

Object detection in security applications using dominant edge directions

https://doi.org/10.1016/j.patrec.2014.09.018Get rights and content

Highlights

  • We introduce “dominant edge directions” (DED) – new image features.

  • DED are based on image edge distribution.

  • We present an application of computer-aided video surveillance – knife detection.

  • We use a combined DED and histograms of oriented gradients (HOG) detector.

  • The combined use of DED and HOG speeds up detection roughly fivefold over HOG alone.

Abstract

This paper presents a novel approach to object detection in images. We build on the existing work on detecting knives in images, which has previously attempted to solve the problem by using the well-established histogram of oriented gradients (HOG) features. We introduce a new feature set that allows for rapid initial object location in images, and can then be followed by the use of an object specific detector. This approach allows for speeding up the overall detection process, which has been demonstrated on the example of knives, and is in the position of bringing many object detectors closer to real-time execution speeds.

Introduction

Computer-aided video surveillance has been receiving significant attention as it enables authorities to trace potential threats to public safety. It can be used both in security applications [1] and as legal evidence [2]. A person's effectiveness simultaneously overlooking multiple screens decreases over time owing to the task's mostly mundane and repetitive nature. One study suggests that detection rates for operators monitoring four, nine and 16 screens oscillate around 83%, 84% and 64% respectively, and will drop significantly after an hour [3]. Although people will almost always outperform software algorithms for object detection in images [4], in the long run the computer could be of significant help to the human CCTV (closed-circuit television) operator when it comes to dealing with tens of simultaneous video feeds for many hours a day. Therefore, the need to automate the process is obvious.

Algorithms analysing video content from surveillance cameras that raise an alarm should a potentially dangerous situation be detected are of great help to the human CCTV operator and security staff in general. Applications for automated computer programs analysing video surveillance material include external intruder detection, people counting and identification, and crowd flux analysis [5]. Another interesting application is detection of dangerous tools such as pistols, batons, etc. In this paper we focus on detecting knives. The importance of detecting such a wide and diverse class of objects is significant. The research presented in this paper aims at creating a robust real-time knife detector that could be used in security applications. It follows the earlier attempts [6] in this area using histogram of oriented gradients [7] as image features.

This paper is organised as follows. After a literature review on computer-aided video surveillance and the description of our research motivation in the remainder of this section, in Section 2 we introduce new image features that highlight the crucial information conveyed in the image through its dominant edge directions. In Section 3 we describe a classifier for the newly introduced features, as well as the datasets that were used for its training and validation. A comparison with other image features is carried out in Section 4, while a detector utilising both dominant edge directions and the famous HOG features [7] is described in Section 5. An experiment on real-life images utilising the combined detector is presented in Section 6. Section 7 concludes the paper with a summarising discussion.

Despite the fact that there have been numerous attempts at detecting suspicious events in video material [8], [9] or recognising human activity in videos [10], [11], [12], to the authors’ best knowledge there has been very little other research on detecting dangerous tools, specifically when it comes to knife detection. Analysis of the currently published literature allows to conclude that there are two distinct categories of dangerous tools, visual detection of which has been considered as an element of more broadly understood computer-aided video surveillance of the public. The first category of dangerous tools is pistols. These have been subject to visual detection attempts in single images or video sequences. Researchers have utilised either feature-based techniques to detect instances of a hand gun appearing in the analysed image, with image features such as SIFT (scale-invariant feature transform) [13] as in [14], or shape-descriptors such as the MPEG-7 shape descriptor [15] as in [16]. Shape analysis is widely used in computerised video-surveillance in applications such as fall detection [17] or more generally as a component of a motion intensity descriptor as in [18]. In addition to specific implementations of pistol detection algorithms there have been a number of research papers published on the subject of whether computerised CCTV analysis can lead to reliable results in general [19], [20]. A fairly broad overview of automated video surveillance techniques has been presented in [21].

When it comes to the other category of dangerous tools – knives – the amount of published research has been rather limited. We are only aware of two research papers, apart from ours [6], [22], that deal with this topic. In [23] the authors approach the problem with the use of Haar cascades [24]. And an interesting approach was used in [25], where the shape of the knife blade is approximated with contours (not edges [26]) and is then classified with a fuzzy classifier [27]. In [22] a novel approach to visual object detection in general was demonstrated using knives, where active appearance models [28] were used for the first time to detect an object rather than locate it.

One limitation of the HOG (histogram of oriented gradients) features that have been used for knife detection [6] is their non-invariance to rotation. Since knives can appear in any angular orientation, the analysed image needs to be rotated by 360° at a fixed angle step to enable detection of objects appearing in arbitrary orientations. Analysis of a single frame of the size 640 × 480 using HOG takes some 83  ms on a modern PC (Intel Core i5 3.10  GHz, 8GB RAM). If we were to rotate that image by 360° at a 15° angle step, the detection procedure would take almost 2  s (0.5 FPS).

This is far from real-time, and probably takes too long for this algorithm to be used in computer-aided video surveillance. One approach that speeds up the relatively long process of running the HOG detector is performing the calculations in a graphical processor unit (GPU). This leads to a very reasonable reduction of the algorithm's execution time. We compared execution times of a serial HOG implementation on the Intel Core i5 machine with the parallel HOG implementation provided in OpenCV [29] using a high-end NVIDIA GeForce GTX 680 graphics card (1536 cores). The results of this comparison are illustrated in Fig. 1.

For a 640 × 480 (0.307 MP) image the improvement was almost fivefold, which means the whole detection procedure involving image rotation takes 0.4–0.5  s (2 FPS). But because a fairly sophisticated graphics card had to be deployed to analyse a single video sequence we looked for ways to speed up the HOG based detection. We now propose a new image feature set that allows initially narrowing the analysed image to an ROI containing objects likely to be knives. Those objects can then be subsequently verified with a HOG detector.

Detecting a knife in an image is challenging. First, knives vary in size, and are small compared to the objects of popular and well- established detection algorithms that find objects such as people [7] and faces [24]. Second, they are not very visible; in the case of a person holding a knife the handle is invisible – the only thing that can be spotted is the blade.

The average knife blade lacks texture and theoretically could be any colour. Therefore methods based on colour detection such as skin detectors as presented in [30] or [31] would be of limited use. The only salient feature it can be distinguished by is its shape, which unfortunately varies greatly among knives. For instance, the average knife has a tip, but the shape of the blade's edges varies from knife to knife as depicted in Fig. 2. One thing all blades do seem to have in common, though, is the dominant edge direction of the blade.

Section snippets

Dominant edge directions image features

Let us assume that we are to detect a knife centred vertically in the detection window. Since the knife blade has a longitudinal shape, detecting the dominant edge direction consistent with the detection window's vertical axis could constitute a possible knife candidate. The proposed image features combine the dominant edge directions (DED) in the image in question with the spatial distribution of the edges in the image. These new features highlight the crucial information that allows for knife

Classifier and datasets

A support vector machine (SVM) with a linear kernel was used as the classifier using the SVM-light implementation [35]. It is common practice to use SVMs with radial basis function kernels in image classification, as typically the best possible predictive performance is better for non-linear kernels [36]. In the course of our experiments, however, we were unable to obtain a better classification accuracy using non-linear SVMs with radial basis function kernels than the classification accuracy

HOG detector

The validation dataset described in Section 3 consisted of 49 64 × 128 images containing knives, and 500 images of this size that did not contain the detection object. In [6] a purely HOG based knife detector is described, so we decided to evaluate the DED and HOG detectors in terms of classification quality and execution times on the validation dataset. Table 2 shows the results of this comparison. When it comes to classifying a single 64 × 128 pixel image, classification quality measured in

Combined DED and HOG detector

The advantage of combining DED and HOG detectors is speeding up of the overall detection process, as, as shown in Section 4, DED's execution time is on average 10 times shorter compared to HOG's for a 512 × 512 pixel image. We propose a detection scheme whose aim is to detect knives in images coming from video surveillance sequences.

In order to set a reasonable ROI on the analysed image, the person appearing in it is first detected. This is done with the HOG detector trained to detect

Experimental results and analysis

An experiment was carried out on a positive and a negative test set composed of 50 images each. The positive test set contained images of a person holding a knife, while in the negative test set no people appeared in the images. For the sake of the experiment, we manually cropped square regions with a person in the middle from images in the positive test set instead of using a people detector, since it could have affected the detection results. The detection procedure described in Section 5 was

Conclusions

This paper proposes new image features to be used in knife detection in images and video sequences. Dominant edge directions (DED) extract information on the spatial distribution of edges and can be used to detect knives. Because the DED based SVM linear classifier is highly sensitive and has relatively low specificity, DED can be used to initially mark potential knife candidates that later undergo verification with a HOG detector trained to detect knives. An undoubted advantage of the knife

Acknowledgement

This work has been co-financed by the European Regional Development Fund under the Innovative Economy Operational Programme, INSIGMA project no. POIG.01.01.02-00-062/09.

References (39)

  • L. Chen et al.

    A survey of human motion analysis using depth imagery

    Pattern Recogn. Lett.

    (2013)
  • M. Kawulok et al.

    Spatial-based skin detection using discriminative skin-presence features

    Pattern Recogn. Lett.

    (2014)
  • M.J. Donahue et al.

    On the use of level curves in image analysis

  • Y. Nam et al.

    Intelligent video surveillance system: 3-tier context-aware surveillance system with metadata

    Multimedia Tools Appl.

    (2012)
  • A. Maalouf et al.

    Offline quality monitoring for legal evidence images in video-surveillance applications

    Multimedia Tools Appl.

    (2012)
  • A.H. Tickner et al.

    Monitoring up to 16 synthetic television pictures showing a great deal of movement

    Ergonomics

    (1973)
  • R. Tadeusiewicz

    How intelligent should be system for image analysis?

  • W. Hu et al.

    A survey on visual surveillance of object motion and behaviors

    IEEE Trans. Syst. Man Cybern.

    (2004)
  • M. Kmieć et al.

    An approach to robust visual knife detection, machine graphics and vision

    Mach. Graph. Vis.

    (2011)
  • N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: CVPR 2005. IEEE Computer Society...
  • G. Lavee et al.

    A framework for a video analysis tool for suspicious event detection

    Multimedia Tools Appl.

    (2007)
  • B. Kaluža, G. Kaminka, M. Tambe, Detection of suspicious behavior from a sparse set of multiagent interactions, in:...
  • Y.-M. Liang et al.

    Human action segmentation and classification based on the Isomap algorithm

    Multimedia Tools Appl.

    (2011)
  • M. Nael, M. Abd El Wahab, M. El-Saban, Multi-view human action recognition system employing 2DPCA, in: WACV,...
  • D. Lowe, Method and apparatus for identifying scale invariant features in an image and use of same for locating an...
  • I.T. Darker, P. Kuo, M.Y. Yang, A. Blechko, C. Grecos, D. Makris, J.-C. Nebel, A.G. Gale, Automation of the...
  • B.S. Manjunath et al.

    Introduction to MPEG-7: Multimedia Content Description Interface

    (2002)
  • M. Grega et al.

    Automated recognition of firearms in surveillance video

  • C. Rougier et al.

    Robust video surveillance for fall detection based on human shape deformation

    IEEE Trans. Circuits Syst. Video Technol.

    (2011)
  • Cited by (26)

    • Computationally convolutional ghost imaging

      2022, Optics and Lasers in Engineering
    • Extended variational inference for gamma mixture model in positive vectors modeling

      2021, Neurocomputing
      Citation Excerpt :

      It is one of the most fundamental yet challenging problem in computer vision (CV) since differing poses, scales, illuminations, and occlusions can incur two images from the same category rather different. Over the past few decades, researchers have made a large number of efforts to cope with this problem, since it has a wide range of applications in various fields such as robot vision [25], public security [26], autonomous vehicles [27], human computer interaction [28], and industrial inspection [29]. These researches can be divided into two categories that one has been devoted to design good handcrafted features [30–32] and the other has focused on designing excellent classifiers [33–35].

    • Brightness guided preprocessing for automatic cold steel weapon detection in surveillance videos with deep learning

      2019, Neurocomputing
      Citation Excerpt :

      The authors in [8] detects a knife in a RGB-images using the sliding window search approach combined with the classical SVM classifier. All these methods provide good accuracies but suffer from several limitations, they are invasive, need expensive metal-detector systems [5] such as the systems used in the airport access, cannot detect multiple weapons [11,26] and are slow to be used in real time detection systems [1]. The state-of-the-art object detection models are based on deep Convolutional Neural Networks and showed promising results in the two most prestigious detection challenges.

    • Study of fluid edge detection and tracking method in glass flume based on image processing technology

      2017, Advances in Engineering Software
      Citation Excerpt :

      With the development of image sensors and image processing technology, measurement methods based on digital image processing have received more and more attention and have been widely used in many engineering studies [1–4].

    View all citing articles on Scopus

    This paper has been recommended for acceptance by R. Davies.

    View full text