Video anomaly detection using CycleGan based on skeleton features

Abstract

Anomaly detection is a challenging task in the field of intelligent video surveillance. It aims to identify anomalous events by monitoring the video captured by visual sensors. The main difficulty of this task is that the definition of anomalies is ambiguous. In recent years, most anomaly detection methods use a two-stage learning strategy, i.e., feature extraction and model building. In this paper, with the idea of refactoring, we propose an end-to-end anomaly detection framework using cyclic consistent adversarial networks (CycleGAN). Dynamic skeleton features are used as network constraints to alleviate the inaccuracy of feature extraction algorithms of a single generative adversarial network. In the training phase, only normal video frames and the corresponding skeleton features are used to train the generator and discriminator. In the testing phase, anomalous behaviors with high reconstruction errors can be filtered out by manually set thresholds. To the best of our knowledge, this is the first time CycleGAN has been used for video anomaly detection. Experimental results on challenging datasets show that our method can accurately detect anomalous behaviors in videos collected by video surveillance systems and is comparable to the current state-of-the-art methods.

Introduction

Video anomaly detection [1], [2], [3], [4] has received more and more research attention around the world for its great significance in real-world applications such as intelligent visual monitoring, traffic detection, and autopilot. However, it is still faced with some challenging problems.

The ambiguous definition of human-perceivable abnormal behavior prevents the optimization of anomaly detection. The same behavior can be recognized as normal or abnormal behavior in different scenarios. For example, ‘running’ is a normal behavior in the sports field, while it should be detected as abnormal activity in a high way. At the same time, there is an imbalance between positive and negative samples in the training dataset, because abnormal activities happen with a much lower probability than normal activities in reality..

Many previous works [5], [6], [7] used pixel-based features or motion features extracted from video frames to train the network. Nevertheless, pixel-based features or motion features extracted from the original frames cover much hidden redundant information, which will increase the difficulty of model training. In addition, pixel-based features are sensitive to noise, leading to the coverage of important and useful information. Human skeleton features, which have rich semantic information and a strong sense of structure, are employed to alleviate the problem. However, the algorithm for skeleton features extraction is not accurate enough to some extent. Taking only skeleton features as input to train the model [5], [8], that is to say, totally depends on the skeleton features, brings about inaccuracy in detection results.

To solve the problems mentioned above, we propose an anomaly detection method. The idea of reconstruction is employed. Only positive samples are fed into the network. CycleGAN [9] is adopted to set up a mapping from the original frame to extracted skeleton features. Using the cycle-consistency loss, two generators and discriminators are trained to minimize the error in reconstructing the video frames and the error in reconstructing the pose estimation map. A manually set threshold is used to filter out the abnormal behavior in the testing phase.

The contribution of our work can be listed as follows.

1. We use only positive samples to train the model, and screen out the abnormal behavior by a preset reconstruction error value, alleviating the problem of ambiguous definition of abnormality and imbalance between positive and negative samples.

2. CycleGAN is applied to anomaly detection in a reconstructive manner, preserving the details of video frames and dynamic skeleton features. To the best of our knowledge, this is the first time CycleGAN is being used in video anomaly detection.

3. Skeleton feature, which has a strong ability to describe the movements of human beings, is taken as the constrain for reconstruction, avoiding the disadvantages of total dependency of skeleton features.

The paper is organized as follows. In Section 2 we review the previous work on anomaly detection. The detail of the proposed method is described in Section 3. Section 4 gives the experimental results and analysis. Section 5 is the conclusion.