Abstract
Log data is typically the only available data source recording system health information. Event extraction converts unstructured log messages into structured event signatures. Existing methods, whether batch or streaming methods, require true event signatures to guide parameter selection. This paper presents a streaming event extraction method that eliminates the demands of external tags and generates appropriate event signatures by evaluating the quality of them. Experimental results show that our approach can parse log message into high-quality information efficiently and detect more anomalies.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Li T, Liu Z, Zhou Q (2016) Application-driven big data mining. ZTE Technol J 22(2):49–52
Fu Q, Lou JG et al (2009) Execution anomaly detection in distributed systems through unstructured log analysis. In: 9th IEEE international conference on data mining. IEEE, pp 149–158
Xu W, Huang L et al (2009) Detecting large-scale system problems by mining console logs. In: 22nd ACM symposium on operating systems principles. ACM, pp 117–132
Nagaraj K, Killian C, Neville J (2012) Structured comparative analysis of systems logs to diagnose performance problems. In: 9th USENIX conference on networked systems design and implementation. USENIX Association, pp 26–26
Ma S, Hellerstein JL (2001) Mining partially periodic event patterns with unknown periods. In: 17th international conference on data engineering. IEEE, pp 205–214
Shang W, Jiang ZM et al (2013) Assisting developers of big data analytics applications when deploying on hadoop clouds. In: 35th international conference on software engineering. IEEE Press, pp 402–411
He P, Zhu J et al (2017) Drain: an online log parsing approach with fixed depth tree. In: 2017 IEEE international conference on web services. IEEE, pp 33–40
Liu Z, Li T, Wang J (2016) A survey on event mining for ICT network infrastructure management. ZTE Commun 14(2):47–55
Lang D (2013) Using sec. USENIX; Login Mag 38(6):38–43
Ning X, Jiang G, Chen H et al (2014) HLAer: a system for heterogeneous log analysis
Terrizzano IG, Schwarz PM et al Data wrangling: the challenging journey from the wild to the lake. In: 7th biennial conference on innovative data systems research
Vaarandi R (2003) A data clustering algorithm for mining patterns from event logs. In: 3th IEEE international workshop IP operations and management. IEEE, pp 119–126
Tang L, Li T, Perng CS (2011) Logsig: generating system events from raw textual logs. In: 20th ACM international conference on information and knowledge management. ACM, pp 785–794
Makanju A, Zincir-Heywood et al (2012) A lightweight algorithm for message type extraction in system application logs. IEEE Trans Knowl Data Eng 24(11):1921–1936
Du M, Li F (2016) Spell: streaming parsing of system event logs. In: 2016 IEEE 16th international conference on data mining. IEEE, pp 859–864
Mizutani M (2013) Incremental mining of system log format. In: 2013 IEEE International Conference on Services Computing. IEEE, pp 595–602
Liu Y, Li Z, Xiong H et al (2010) Understanding of internal clustering validation measures. In: 10th international conference on data mining. IEEE, pp 911–916
Acknowledgements
This work is supported in part by Jiangsu Provincial Natural Science Foundation of China under Grant BK20171447, Jiangsu Provincial University Natural Science Research of China under Grant 17KJB520024, and Nanjing University of Posts and Telecommunications under Grant No. NY215045.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Guo, S., Liu, Z., Chen, W., Li, T. (2019). Event Extraction from Streaming System Logs. In: Kim, K., Baek, N. (eds) Information Science and Applications 2018. ICISA 2018. Lecture Notes in Electrical Engineering, vol 514. Springer, Singapore. https://doi.org/10.1007/978-981-13-1056-0_47
Download citation
DOI: https://doi.org/10.1007/978-981-13-1056-0_47
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-1055-3
Online ISBN: 978-981-13-1056-0
eBook Packages: EngineeringEngineering (R0)