Object detection in security applications using dominant edge directions
Introduction
Computer-aided video surveillance has been receiving significant attention as it enables authorities to trace potential threats to public safety. It can be used both in security applications [1] and as legal evidence [2]. A person's effectiveness simultaneously overlooking multiple screens decreases over time owing to the task's mostly mundane and repetitive nature. One study suggests that detection rates for operators monitoring four, nine and 16 screens oscillate around 83%, 84% and 64% respectively, and will drop significantly after an hour [3]. Although people will almost always outperform software algorithms for object detection in images [4], in the long run the computer could be of significant help to the human CCTV (closed-circuit television) operator when it comes to dealing with tens of simultaneous video feeds for many hours a day. Therefore, the need to automate the process is obvious.
Algorithms analysing video content from surveillance cameras that raise an alarm should a potentially dangerous situation be detected are of great help to the human CCTV operator and security staff in general. Applications for automated computer programs analysing video surveillance material include external intruder detection, people counting and identification, and crowd flux analysis [5]. Another interesting application is detection of dangerous tools such as pistols, batons, etc. In this paper we focus on detecting knives. The importance of detecting such a wide and diverse class of objects is significant. The research presented in this paper aims at creating a robust real-time knife detector that could be used in security applications. It follows the earlier attempts [6] in this area using histogram of oriented gradients [7] as image features.
This paper is organised as follows. After a literature review on computer-aided video surveillance and the description of our research motivation in the remainder of this section, in Section 2 we introduce new image features that highlight the crucial information conveyed in the image through its dominant edge directions. In Section 3 we describe a classifier for the newly introduced features, as well as the datasets that were used for its training and validation. A comparison with other image features is carried out in Section 4, while a detector utilising both dominant edge directions and the famous HOG features [7] is described in Section 5. An experiment on real-life images utilising the combined detector is presented in Section 6. Section 7 concludes the paper with a summarising discussion.
Despite the fact that there have been numerous attempts at detecting suspicious events in video material [8], [9] or recognising human activity in videos [10], [11], [12], to the authors’ best knowledge there has been very little other research on detecting dangerous tools, specifically when it comes to knife detection. Analysis of the currently published literature allows to conclude that there are two distinct categories of dangerous tools, visual detection of which has been considered as an element of more broadly understood computer-aided video surveillance of the public. The first category of dangerous tools is pistols. These have been subject to visual detection attempts in single images or video sequences. Researchers have utilised either feature-based techniques to detect instances of a hand gun appearing in the analysed image, with image features such as SIFT (scale-invariant feature transform) [13] as in [14], or shape-descriptors such as the MPEG-7 shape descriptor [15] as in [16]. Shape analysis is widely used in computerised video-surveillance in applications such as fall detection [17] or more generally as a component of a motion intensity descriptor as in [18]. In addition to specific implementations of pistol detection algorithms there have been a number of research papers published on the subject of whether computerised CCTV analysis can lead to reliable results in general [19], [20]. A fairly broad overview of automated video surveillance techniques has been presented in [21].
When it comes to the other category of dangerous tools – knives – the amount of published research has been rather limited. We are only aware of two research papers, apart from ours [6], [22], that deal with this topic. In [23] the authors approach the problem with the use of Haar cascades [24]. And an interesting approach was used in [25], where the shape of the knife blade is approximated with contours (not edges [26]) and is then classified with a fuzzy classifier [27]. In [22] a novel approach to visual object detection in general was demonstrated using knives, where active appearance models [28] were used for the first time to detect an object rather than locate it.
One limitation of the HOG (histogram of oriented gradients) features that have been used for knife detection [6] is their non-invariance to rotation. Since knives can appear in any angular orientation, the analysed image needs to be rotated by 360° at a fixed angle step to enable detection of objects appearing in arbitrary orientations. Analysis of a single frame of the size 640 × 480 using HOG takes some 83 ms on a modern PC (Intel Core i5 3.10 GHz, 8GB RAM). If we were to rotate that image by 360° at a 15° angle step, the detection procedure would take almost 2 s (0.5 FPS).
This is far from real-time, and probably takes too long for this algorithm to be used in computer-aided video surveillance. One approach that speeds up the relatively long process of running the HOG detector is performing the calculations in a graphical processor unit (GPU). This leads to a very reasonable reduction of the algorithm's execution time. We compared execution times of a serial HOG implementation on the Intel Core i5 machine with the parallel HOG implementation provided in OpenCV [29] using a high-end NVIDIA GeForce GTX 680 graphics card (1536 cores). The results of this comparison are illustrated in Fig. 1.
For a 640 × 480 (0.307 MP) image the improvement was almost fivefold, which means the whole detection procedure involving image rotation takes 0.4–0.5 s (2 FPS). But because a fairly sophisticated graphics card had to be deployed to analyse a single video sequence we looked for ways to speed up the HOG based detection. We now propose a new image feature set that allows initially narrowing the analysed image to an ROI containing objects likely to be knives. Those objects can then be subsequently verified with a HOG detector.
Detecting a knife in an image is challenging. First, knives vary in size, and are small compared to the objects of popular and well- established detection algorithms that find objects such as people [7] and faces [24]. Second, they are not very visible; in the case of a person holding a knife the handle is invisible – the only thing that can be spotted is the blade.
The average knife blade lacks texture and theoretically could be any colour. Therefore methods based on colour detection such as skin detectors as presented in [30] or [31] would be of limited use. The only salient feature it can be distinguished by is its shape, which unfortunately varies greatly among knives. For instance, the average knife has a tip, but the shape of the blade's edges varies from knife to knife as depicted in Fig. 2. One thing all blades do seem to have in common, though, is the dominant edge direction of the blade.
Section snippets
Dominant edge directions image features
Let us assume that we are to detect a knife centred vertically in the detection window. Since the knife blade has a longitudinal shape, detecting the dominant edge direction consistent with the detection window's vertical axis could constitute a possible knife candidate. The proposed image features combine the dominant edge directions (DED) in the image in question with the spatial distribution of the edges in the image. These new features highlight the crucial information that allows for knife
Classifier and datasets
A support vector machine (SVM) with a linear kernel was used as the classifier using the SVM-light implementation [35]. It is common practice to use SVMs with radial basis function kernels in image classification, as typically the best possible predictive performance is better for non-linear kernels [36]. In the course of our experiments, however, we were unable to obtain a better classification accuracy using non-linear SVMs with radial basis function kernels than the classification accuracy
HOG detector
The validation dataset described in Section 3 consisted of 49 64 × 128 images containing knives, and 500 images of this size that did not contain the detection object. In [6] a purely HOG based knife detector is described, so we decided to evaluate the DED and HOG detectors in terms of classification quality and execution times on the validation dataset. Table 2 shows the results of this comparison. When it comes to classifying a single 64 × 128 pixel image, classification quality measured in
Combined DED and HOG detector
The advantage of combining DED and HOG detectors is speeding up of the overall detection process, as, as shown in Section 4, DED's execution time is on average 10 times shorter compared to HOG's for a 512 × 512 pixel image. We propose a detection scheme whose aim is to detect knives in images coming from video surveillance sequences.
In order to set a reasonable ROI on the analysed image, the person appearing in it is first detected. This is done with the HOG detector trained to detect
Experimental results and analysis
An experiment was carried out on a positive and a negative test set composed of 50 images each. The positive test set contained images of a person holding a knife, while in the negative test set no people appeared in the images. For the sake of the experiment, we manually cropped square regions with a person in the middle from images in the positive test set instead of using a people detector, since it could have affected the detection results. The detection procedure described in Section 5 was
Conclusions
This paper proposes new image features to be used in knife detection in images and video sequences. Dominant edge directions (DED) extract information on the spatial distribution of edges and can be used to detect knives. Because the DED based SVM linear classifier is highly sensitive and has relatively low specificity, DED can be used to initially mark potential knife candidates that later undergo verification with a HOG detector trained to detect knives. An undoubted advantage of the knife
Acknowledgement
This work has been co-financed by the European Regional Development Fund under the Innovative Economy Operational Programme, INSIGMA project no. POIG.01.01.02-00-062/09.
References (39)
- et al.
A survey of human motion analysis using depth imagery
Pattern Recogn. Lett.
(2013) - et al.
Spatial-based skin detection using discriminative skin-presence features
Pattern Recogn. Lett.
(2014) - et al.
On the use of level curves in image analysis
- et al.
Intelligent video surveillance system: 3-tier context-aware surveillance system with metadata
Multimedia Tools Appl.
(2012) - et al.
Offline quality monitoring for legal evidence images in video-surveillance applications
Multimedia Tools Appl.
(2012) - et al.
Monitoring up to 16 synthetic television pictures showing a great deal of movement
Ergonomics
(1973) How intelligent should be system for image analysis?
- et al.
A survey on visual surveillance of object motion and behaviors
IEEE Trans. Syst. Man Cybern.
(2004) - et al.
An approach to robust visual knife detection, machine graphics and vision
Mach. Graph. Vis.
(2011) - N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: CVPR 2005. IEEE Computer Society...
A framework for a video analysis tool for suspicious event detection
Multimedia Tools Appl.
Human action segmentation and classification based on the Isomap algorithm
Multimedia Tools Appl.
Introduction to MPEG-7: Multimedia Content Description Interface
Automated recognition of firearms in surveillance video
Robust video surveillance for fall detection based on human shape deformation
IEEE Trans. Circuits Syst. Video Technol.
Cited by (26)
An edge detail enhancement strategy based on Fourier single-pixel imaging
2024, Optics and Lasers in EngineeringComputationally convolutional ghost imaging
2022, Optics and Lasers in EngineeringExtended variational inference for gamma mixture model in positive vectors modeling
2021, NeurocomputingCitation Excerpt :It is one of the most fundamental yet challenging problem in computer vision (CV) since differing poses, scales, illuminations, and occlusions can incur two images from the same category rather different. Over the past few decades, researchers have made a large number of efforts to cope with this problem, since it has a wide range of applications in various fields such as robot vision [25], public security [26], autonomous vehicles [27], human computer interaction [28], and industrial inspection [29]. These researches can be divided into two categories that one has been devoted to design good handcrafted features [30–32] and the other has focused on designing excellent classifiers [33–35].
Brightness guided preprocessing for automatic cold steel weapon detection in surveillance videos with deep learning
2019, NeurocomputingCitation Excerpt :The authors in [8] detects a knife in a RGB-images using the sliding window search approach combined with the classical SVM classifier. All these methods provide good accuracies but suffer from several limitations, they are invasive, need expensive metal-detector systems [5] such as the systems used in the airport access, cannot detect multiple weapons [11,26] and are slow to be used in real time detection systems [1]. The state-of-the-art object detection models are based on deep Convolutional Neural Networks and showed promising results in the two most prestigious detection challenges.
Multimodal architecture for video captioning with memory networks and an attention mechanism
2018, Pattern Recognition LettersStudy of fluid edge detection and tracking method in glass flume based on image processing technology
2017, Advances in Engineering SoftwareCitation Excerpt :With the development of image sensors and image processing technology, measurement methods based on digital image processing have received more and more attention and have been widely used in many engineering studies [1–4].
This paper has been recommended for acceptance by R. Davies.