Fan Yang
Bin Guo
Skip Abstract Section
Abstract
It is still a challenge to detect anomalous events in video sequences in the field of computer vision due to heavy object occlusions, varying crowded densities and complex situations. To address this, we propose a novel human-machine cooperative approach which uses human feedback on anomaly confirmation to inform and enhance video anomaly detection. Specifically, we analyze the spatio-temporal characteristics of sequential frames of a video from the appearance and motion perspective from which spatial and temporal features are identified and extracted. We then develop a convolutional autoencoder neural network to compute an abnormal score based on reconstruction errors. In this process, a group of experts will provide human feedback to a certain proportion of classified frames to be incorporated into the model, and also the final judgment for the event anomalies for training and classification. The proposed approach is evaluated on 3 publicly available surveillance datasets, showing improved accuracy and competitive performance (93.7% AUC) with respect to the best performance (90.6% AUC) of the state-of-the-art approaches. The approach has not been previously seen to the best of our knowledge.
- Amit Adam, Ehud Rivlin, Ilan Shimshoni, and Daviv Reinitz. 2008. Robust real-time unusual event detection using multiple fixed-location monitors. IEEE transactions on pattern analysis and machine intelligence 30, 3 (2008), 555--560. https://doi.org/10.1109/TPAMI.2007.70825Google Scholar
Digital Library
- Saleema Amershi, Maya Cakmak, W. Bradley Knox, and Todd Kulesza. 2014. Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine 35, 4 (2014), 105--120. https://doi.org/10.1609/aimag.v35i4.2513Google ScholarDigital Library
- Saleema Amershi, James Fogarty, and Daniel Weld. 2012. ReGroup: Interactive Machine Learning for On-Demand Group Creation in Social Networks. In Proceeding CHI '12 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 21--30. https://doi.org/10.1145/2207676.2207680Google ScholarDigital Library
- Y. Benezeth, P. Jodoin, V. Saligrama, and C. Rosenberger. 2009. Abnormal events detection based on spatio-temporal co-occurences. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2458--2465. https://doi.org/10. 1109/CVPR.2009.5206686Google ScholarCross Ref
- Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. 2019. Human-centered tools for coping with imperfect algorithms during medical decision-making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1--14. https://doi.org/10.1145/3290605.3300234Google ScholarDigital Library
- Karanbir Singh Chahal and Kuntal Dey. 2018. A Survey of Modern Object Detection Literature using Deep Learning. ArXiv abs/1808.07256 (2018). arXiv:1808.07256 http://arxiv.org/abs/1808.07256Google Scholar
- Rima Chaker, Zaher Al Aghbari, and Imran N. Junejo. 2017. Social network model for crowd anomaly detection and localization. Pattern Recognition 61 (2017), 266--281. https://doi.org/10.1016/j.patcog.2016.06.016Google ScholarDigital Library
- Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Comput. Surv. 41, 3 (2009), 15:1--15:58. https://doi.org/10.1145/1541880.1541882Google ScholarDigital Library
- Justin Cheng and Michael S Bernstein. 2015. Flock: Hybrid crowd-machine learning classifiers. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. New York, NY, USA, 600--611. https://doi.org/10.1145/2675133.2675214Google ScholarDigital Library
- Yong Shean Chong and Yong Haur Tay. 2017. Abnormal event detection in videos using spatiotemporal autoencoder. In International Symposium on Neural Networks. Springer, 189--196. https://doi.org/10.1007/978-3-319-59081-3_23Google ScholarCross Ref
- Rensso Victor Hugo Mora Colque, Carlos Caetano, Matheus Toledo Lustosa de Andrade, and William Robson Schwartz. 2017. Histograms of Optical Flow Orientation and Magnitude and Entropy to Detect Anomalous Events in Videos. IEEE Trans. Circuits Syst. Video Techn. 27, 3 (2017), 673--682. https://doi.org/10.1109/TCSVT.2016.2637778Google ScholarDigital Library
- Yang Cong, Junsong Yuan, and Ji Liu. 2011. Sparse Reconstruction Cost for Abnormal Event Detection. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, USA, 3449--3456. https://doi.org/10.1109/CVPR.2011.5995434Google ScholarDigital Library
- Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In IEEE computer society conference on computer vision and pattern recognition (CVPR'05), Vol. 1. IEEE, 886--893. https://doi.org/10.1109/CVPR. 2005.177Google ScholarCross Ref
- Jerry Alan Fails and Dan R Olsen Jr. 2003. Interactive machine learning. In Proceedings of the 8th international conference on Intelligent user interfaces. ACM, 39--45. https://doi.org/10.1145/604045.604056Google ScholarDigital Library
- James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: Interactive Concept Learning in Image Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI '08). Association for Computing Machinery, New York, NY, USA, 29--38. https://doi.org/10.1145/1357054.1357061Google ScholarDigital Library
- Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K Roy-Chowdhury, and Larry S Davis. 2016. Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition. 733--742. https://doi.org/10.1109/CVPR.2016.86Google ScholarCross Ref
- Andreas Holzinger, Markus Plass, Michael Kickmeier-Rust, Katharina Holzinger, Gloria Cerasela Crişan, Camelia-M Pintea, and Vasile Palade. 2019. Interactive machine learning: experimental evidence for the human in the algorithmic loop. Applied Intelligence 49, 7 (2019), 2401--2414. https://doi.org/10.1007/s10489-018-1361-5Google ScholarDigital Library
- Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2012. 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence 35, 1 (2012), 221--231. https://doi.org/10.1109/TPAMI.2012.59Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105. https://doi.org/10.1145/3065386Google ScholarDigital Library
- Teng Lee, James Johnson, and Steve Cheng. 2016. An Interactive Machine Learning Framework. arXiv preprint arXiv:1610.05463 abs/1610.05463 (2016). arXiv:1610.05463 http://arxiv.org/abs/1610.05463Google Scholar
- Weixin Li, Vijay Mahadevan, and Nuno Vasconcelos. 2013. Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence 36, 1 (2013), 18--32.Google Scholar
- Cewu Lu, Jianping Shi, and Jiaya Jia. 2013. Abnormal Event Detection at 150 FPS in MATLAB. In Proceedings of the 2013 IEEE International Conference on Computer Vision (ICCV '13). IEEE Computer Society, USA, 2720--2727. https://doi.org/10.1109/ICCV.2013.338Google ScholarDigital Library
- Vijay Mahadevan, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos. 2010. Anomaly detection in crowded scenes. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 1975--1981. https: //doi.org/10.1109/CVPR.2010.5539872Google ScholarCross Ref
- Ramin Mehran, Alexis Oyama, and Mubarak Shah. 2009. Abnormal crowd behavior detection using social force model. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 935--942. https://doi.org/10.1109/CVPR. 2009.5206641Google ScholarCross Ref
- Vignesh Ramanathan, Jonathan Huang, Sami Abu-El-Haija, Alexander Gorban, Kevin Murphy, and Li Fei-Fei. 2016. Detecting events and key actors in multi-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3043--3053. https://doi.org/10.1109/CVPR.2016.332Google ScholarCross Ref
- Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2017. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 6 (2017), 1137--1149. https://doi.org/10.1109/TPAMI.2016.2577031Google ScholarDigital Library
- Mohammad Sabokrou, Mahmood Fathy, Mojtaba Hoseini, and Reinhard Klette. 2015. Real-time anomaly detection and localization in crowded scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 56--62. https://doi.org/10.1109/CVPRW.2015.7301284Google ScholarCross Ref
- Dominik Sacha, Michael Sedlmair, Leishi Zhang, John Lee, Jaakko Peltonen, Daniel Weiskopf, Stephen North, and Daniel Keim. 2017. What You See Is What You Can Change: Human-Centered Machine Learning By Interactive Visualization. Neuro computing 268 (04 2017). https://doi.org/10.1016/j.neucom.2017.01.105Google ScholarDigital Library
- Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of the 25th International Conference on Machine Learning (Helsinki, Finland) (ICML '08). ACM, New York, NY, USA, 1096--1103. https://doi.org/10.1145/1390156.1390294Google ScholarDigital Library
- J. Wang, Y. Wang, and Q. Lv. 2019. Crowd-Assisted Machine Learning: Current Issues and Future Directions. Computer 52, 1 (2019), 46--53. https://doi.org/10.1109/MC.2018.2890174Google ScholarDigital Library
- Shandong Wu, Brian E Moore, and Mubarak Shah. 2010. Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2054--2060. https://doi.org/10.1109/CVPR.2010.5539882Google ScholarCross Ref
- Doris Xin, Litian Ma, Jialin Liu, Stephen Macke, Shuchen Song, and Aditya G. Parameswaran. 2018. Accelerating Human-in-the-loop Machine Learning: Challenges and Opportunities. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning, DEEM@SIGMOD 2018, Houston, TX, USA, June 15, 2018. 9:1--9:4. https://doi.org/10.1145/3209889.3209897Google ScholarDigital Library
- Dan Xu, Yan Yan, Elisa Ricci, and Nicu Sebe. 2017. Detecting Anomalous Events in Videos by Learning Deep Representations of Appearance and Motion. Computer Vision and Image Understanding 156 (March 2017), 117--127. https://doi.org/10.1016/j.cviu.2016.10.010Google ScholarDigital Library
- Yiru Zhao, Bing Deng, Chen Shen, Yao Liu, Hongtao Lu, and Xian-Sheng Hua. 2017. Spatio-temporal autoencoder for video anomaly detection. In Proceedings of the 25th ACM international conference on Multimedia. ACM, 1933--1941. https://doi.org/10.1145/3123266.3123451Google ScholarDigital Library
- Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. 2019. Object Detection With Deep Learning: A Review. IEEE Trans. Neural Netw. Learning Syst. 30, 11 (2019), 3212--3232. https://doi.org/10.1109/TNNLS.2018.2876865Google ScholarCross Ref
- Chong Zhou and Randy C. Paffenroth. 2017. Anomaly Detection with Robust Deep Autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13-17, 2017. 665--674. https://doi.org/10.1145/3097983.3098052Google ScholarDigital Library
Index Terms
- Human-Machine Cooperative Video Anomaly Detection
Recommendations
Autoencoding Binary Classifiers for Supervised Anomaly Detection
PRICAI 2019: Trends in Artificial IntelligenceAbstractWe propose the Autoencoding Binary Classifiers (ABC), a novel supervised anomaly detector based on the Autoencoder (AE). There are two main approaches in anomaly detection: supervised and unsupervised. The supervised approach accurately detects ...
Reconstruct Anomaly to Normal: Adversarially Learned and Latent Vector-Constrained Autoencoder for Time-Series Anomaly Detection
PRICAI 2021: Trends in Artificial IntelligenceAbstractTime-series Anomaly Detection has important applications, such as credit card fraud detection and machine fault detection. Anomaly detection based on the generative model generally detect samples with high reconstruction errors as anomalies. ...
Human-machine interactive streaming anomaly detection by online self-adaptive forest
AbstractAnomaly detectors are used to distinguish differences between normal and abnormal data, which are usually implemented by evaluating and ranking the anomaly scores of each instance. A static unsupervised streaming anomaly detector is difficult to ...
Comments