Skip to main content
SpringerLink
Log in

Object and patch based anomaly detection and localization in crowded scenes

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Detecting and localizing anomalies in crowded scenes is an ongoing challenge for public security. Existing approaches are mainly based on patches and trajectories. However, they fall short in semantic understanding of scenes and tackling the depth-of-field problem, respectively. In this paper, we put forward a novel object and patch based framework for anomaly detection and localization. Specifically, we propose to colorize images for precise object detection in dim scenarios. Categories of the objects are used for appearance anomaly justification. For motion anomaly, we propose a new patch-based algorithm which is robust to the depth-of-field problem, which can also be used to detect location anomalies. Besides, a new object re-targeting method is proposed to find the missing objects in detection. It can also handle drift and occlusions in tracking, which can avoid false alarms. Extensive experiments are conducted on several benchmark datasets for anomaly detection. The results show that the proposed method can achieve comparable accuracy in anomaly detection with state-of-the-arts methods and at the same time, localize anomalies precisely.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://drive.google.com/open?id=0B4ucRlpkNQn-Wm8ydWZJc0kyX0E

  2. http://www.svcl.ucsd.edu/projects/anomaly/dataset.html

  3. http://home.ustc.edu.cn/%7Elxd1030/

References

  1. Antonakaki P, Kosmopoulos D, Perantonis SJ (2009) Detecting abnormal human behaviour using multiple cameras. Signal Process 89(9):1723–1738

    Article  MATH  Google Scholar 

  2. Calderara S, Heinemann U, Prati A, Cucchiara R, Tishby N (2011) Detecting anomalies in people’s trajectories using spectral graph analysis. Comput Vis Image Underst 115(8):1099–1111

    Article  Google Scholar 

  3. Cong Y, Yuan J, Liu J (2013) Abnormal event detection in crowded scenes using sparse representation. Pattern Recogn 46(7):1851–1864

    Article  Google Scholar 

  4. Coşar S, Donatiello G, Bogorny V, Garate C, Alvares LO, Brémond F (2017) Toward abnormal trajectory and event detection in video surveillance. IEEE Trans Circuits Syst Video Technol 27(3):683–695

    Article  Google Scholar 

  5. Dai J, Li Y, He K, Sun J (2016) R-fcn: object detection via region-based fully convolutional networks. In: Advances in neural information processing systems, pp 379–387

  6. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005. CVPR 2005. IEEE computer society conference on computer vision and pattern recognition, vol 1. IEEE, pp 886– 893

  7. Dalal N, Triggs B, Schmid C (2006) Human detection using oriented histograms of flow and appearance. In: European conference on computer vision, Springer, pp 428–441

  8. Feng Y, Yuan Y, Lu X (2017) Learning deep event models for crowd anomaly detection. Neurocomputing 219:548–556

    Article  Google Scholar 

  9. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742

  10. He Z, Li X, You X, Tao D, Tang YY (2016) Connected component model for multi-object tracking. IEEE Trans Image Process 25(8):3698–3711

    Article  MathSciNet  MATH  Google Scholar 

  11. Henriques JF, Caseiro R, Martins P, Batista J (2015) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596

    Article  Google Scholar 

  12. Hinami R, Mei T, Satoh S (2017) Joint detection and recounting of abnormal events by learning deep generic knowledge. In: ICCV, pp 3639–3647

  13. Hu W, Xiao X, Fu Z, Xie D, Tan T, Maybank S (2006) A system for learning statistical motion patterns. IEEE Trans Pattern Anal Mach Intell 28(9):1450–1464

    Article  Google Scholar 

  14. Jiang F, Yuan J, Tsaftaris SA, Katsaggelos AK (2011) Anomalous video event detection using spatiotemporal context. Comput Vis Image Underst 115 (3):323–333

    Article  Google Scholar 

  15. Khalid S (2010) Activity classification and anomaly detection using m-mediods based modelling of motion patterns. Pattern Recogn 43(10):3636–3647

    Article  MATH  Google Scholar 

  16. Kim J, Grauman K (2009) Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates. In: 2009. CVPR 2009. IEEE conference on computer vision and pattern recognition, IEEE, pp 2921–2928

  17. Lee S, Kim HG, Ro YM (2018) Stan: spatio-temporal adversarial networks for abnormal event detection. arXiv:1804.08381

  18. Li Q, Li W (2016) A novel framework for anomaly detection in video surveillance using multi-feature extraction. In: 2016 9th international symposium on computational intelligence and design (ISCID), vol 1. IEEE, pp 455–459

  19. Liu Q, Liu B, Yu N (2017) Tccf: tracking based on convolutional neural network and correlation filters. In: International conference on image and graphics, Springer, pp 316–327

  20. Liu W, Luo W, Lian D, Gao S (2017) Future frame prediction for anomaly detection–a new baseline. arXiv:1712.09867

  21. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE international conference on computer vision, pp 2720–2727

  22. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. ICCV 1(2):3

    Google Scholar 

  23. Mahadevan V, Li W, Bhalodia V, Vasconcelos N (2010) Anomaly detection in crowded scenes. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 1975–1981

  24. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: 2009. CVPR 2009. IEEE conference on computer vision and pattern recognition, IEEE, pp 935–942

  25. Morris BT, Trivedi MM (2011) Trajectory learning for activity understanding: unsupervised, multilevel, and long-term adaptive approach. IEEE Trans Pattern Anal Mach Intell 33(11):2287–2301

    Article  Google Scholar 

  26. Piciarelli C, Foresti GL (2006) On-line trajectory clustering for anomalous events detection. Pattern Recogn Lett 27(15):1835–1842

    Article  Google Scholar 

  27. Piciarelli C, Micheloni C, Foresti GL (2008) Trajectory-based anomalous event detection. IEEE Trans Circuits Syst Video Technol 18(11):1544–1554

    Article  Google Scholar 

  28. Ravanbakhsh M, Nabi M, Sangineto E, Marcenaro L, Regazzoni C, Sebe N (2017) Abnormal event detection in videos using generative adversarial nets. In: 2017 IEEE international conference on image processing (ICIP), IEEE, pp 1577–1581

  29. Sabokrou M, Fathy M, Hoseini M, Klette R (2015) Real-time anomaly detection and localization in crowded scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 56–62

  30. Sabokrou M, Fayyaz M, Fathy M, Klette R (2017) Deep-cascade: cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans Image Process 26(4):1992–2004

    Article  MathSciNet  MATH  Google Scholar 

  31. Sabokrou M, Fayyaz M, Fathy M, Moayed Z, Klette R (2018) Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Computer Vision and Image Understanding

  32. Tung F, Zelek JS, Clausi DA (2011) Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis Comput 29(4):230–240

    Article  Google Scholar 

  33. Wang Q, Chen M, Nie F, Li X (2018) Detecting coherent groups in crowd scenes by multiview clustering. IEEE transactions on pattern analysis and machine intelligence

  34. Wang Q, Wan J, Yuan Y (2018) Deep metric learning for crowdedness regression. IEEE Trans Circuits Syst Video Technol 28(10):2633–2643

    Article  Google Scholar 

  35. Xiao T, Zhang C, Zha H (2015) Learning to detect anomalies in surveillance video. IEEE Signal Process Lett 22(9):1477–1481

    Article  Google Scholar 

  36. Xu D, Song R, Wu X, Li N, Feng W, Qian H (2014) Video anomaly detection based on a hierarchical activity discovery within spatio-temporal contexts. Neurocomputing 143:144–152

    Article  Google Scholar 

  37. Xu D, Yan Y, Ricci E, Sebe N (2017) Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput Vis Image Underst 156:117–127

    Article  Google Scholar 

  38. Yan C, Xie H, Chen J, Zha Z, Hao X, Zhang Y, Dai Q (2018) Cross-modality bridging and knowledge transferring for image understanding. IEEE Transactions on Multimedia

  39. Yan C, Xie H, Chen J, Zha Z, Hao X, Zhang Y, Dai Q (2018) A fast uyghur text detector for complex background images. IEEE Trans Multimedia 20 (12):3389–3398. https://doi.org/10.1109/TMM.2018.2838320

    Article  Google Scholar 

  40. Zhang Y, Lu H, Zhang L, Ruan X (2016) Combining motion and appearance cues for anomaly detection. Pattern Recogn 51:443–452

    Article  Google Scholar 

Download references

Acknowledgements

We thank Mahmudul Hasan, W.Luo, W.Liu for sharing their codes and data. This work is supported by the National Natural Science Foundation of China (Grant No. 61371192), the Key Laboratory Foundation of the Chinese Academy of Sciences (CXJJ-17S044) and the Fundamental Research Funds for the Central Universities (WK2100330002, WK3480000005).

Author information

Authors and Affiliations

  1. School of Information Science and Technology, University of Science and Technology of China, Hefei, China

    Xiaodan Li, Weihai Li, Bin Liu & Nenghai Yu

  2. Key Laboratory of Electromagnetic Space Information, Chinese Academy of Sciences, Hefei, China

    Xiaodan Li, Weihai Li, Bin Liu & Nenghai Yu

Authors
  1. Xiaodan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weihai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nenghai Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weihai Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Li, W., Liu, B. et al. Object and patch based anomaly detection and localization in crowded scenes. Multimed Tools Appl 78, 21375–21390 (2019). https://doi.org/10.1007/s11042-019-7447-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7447-1

Keywords

Search

Navigation