Vibhuti Gupta
ABSTRACT
In today's digital era, data are everywhere from Internet of Things to health care or financial applications. This leads to potentially unbounded ever-growing Big data streams and it needs to be utilized effectively. Data normalization is an important preprocessing technique for data analytics. It helps prevent mismodeling and reduce the complexity inherent in the data especially for data integrated from multiple sources and contexts. Normalization of Big Data stream is challenging because of evolving inconsistencies, time and memory constraints, and non-availability of whole data beforehand. This paper proposes a distributed approach to adaptive normalization for Big data stream. Using sliding windows of fixed size, it provides a simple mechanism to adapt the statistics for normalizing changing data in each window. Implemented on Apache Storm, a distributed real-time stream data framework, our approach exploits distributed data processing for efficient normalization. Unlike other existing adaptive approaches that normalize data for a specific use (e.g., classification), ours does not. Moreover, our adaptive mechanism allows flexible controls, via user-specified thresholds, for normalization tradeoffs between time and precision. The paper illustrates our proposed approach along with a few other techniques and experiments on both synthesized and real-world data. The normalized data obtained from our proposed approach, on 160,000 instances of data stream, improves over the baseline by 89% with 0.0041 root-mean-square error compared with the actual data.
- Elwell, R., & Polikar, R., "Incremental learning of concept drift in nonstationary environments", IEEE Transactions on Neural Networks, pp. 1517--1531, 2011.Google Scholar
Digital Library
- García, S., et al., "Tutorial on practical tips of the most influential data preprocessing algorithms in data mining", Knowledge-Based Systems, 98, 1--29, 2016.Google ScholarDigital Library
- García, S., et al., "Big data preprocessing: methods and prospects", Big Data Analytics, 1(1), 9, 2016.Google ScholarCross Ref
- García, S., et al., "Data Preprocessing in Data Mining", Springer, 2015.Google Scholar
- Gu, X. F., et al., "An improving online accuracy updated ensemble method in learning from evolving data streams", In Proceedings of 11th International Computer Conference on Wavelet Actiev Media Technology and Information Processing (ICCWAMTIP), pp. 430--433, 2014.Google ScholarCross Ref
- Han, J., et al., "Data mining: concepts and techniques", San Francisco: Morgan Kauffman, 2001.Google Scholar
- Haykin, S., et al., "Neural networks and learning machines", Upper Saddle River: Pearson education, 2009.Google Scholar
- Hu, H., & Kantardzic, M.," Smart preprocessing improves data stream mining", In Proceedings of 49th Hawaii International Conference on System Sciences (HICSS), pp. 1749--1757, 2016.Google ScholarDigital Library
- Lin, J., & Keogh, E., "Finding or not finding rules in time series", In Applications of Artificial Intelligence in Finance and Economics, pp. 175--201, Emerald Group Publishing Limited, 2004.Google ScholarCross Ref
- Lopez, M. A., et al., "A fast unsupervised preprocessing method for network monitoring", Annals of Telecommunications, 74(3-4), 139--155, 2019.Google ScholarCross Ref
- Ogasawara, E., et al., "Adaptive normalization: A novel data normalization approach for non-stationary time series", In Proceedings of International Joint Conference on Neural Networks (IJCNN), pp. 1--8, 2010.Google Scholar
- Parker, B. S., et al., "Incremental ensemble classifier addressing non-stationary fast data streams", In Proceedings of IEEE International Conference on Data Mining Workshop, pp. 716--723, 2014.Google ScholarCross Ref
- Passalis, N., et al. "Deep Adaptive Input Normalization for Price Forecasting using Limit Order Book Data." arXiv:190.07892, 2019.Google Scholar
- Pyle, D., Data preparation for data mining, morgan kaufmann, 1999.Google ScholarDigital Library
- Ramírez-Gallego, et al., "A survey on data preprocessing for data stream mining: Current status and future directions", Neurocomputing, 239, 39--57, 2017.Google ScholarDigital Library
- Street, W. N., & Kim, Y., "A streaming ensemble algorithm (SEA) for large-scale classification", In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 377--382, 2001.Google ScholarDigital Library
- Tan, P. N., et al., "Association analysis: basic concepts and algorithms", In Introduction to Data mining (Vol. 321321367). Boston, MA: Addison-Wesley, 2005.Google Scholar
- Zliobaite, I., & Gabrys, B., "Adaptive preprocessing for streaming data", IEEE transactions on knowledge and data Engineering, 26(2), 309--321, 2012.Google Scholar
- Toshniwal, Ankit, et al., "Storm@twitter," In Proceedings of the ACM SIGMOD international conference on Management of data, ACM, 2014.Google Scholar
- Harries, M., & Wales, N. S., Splice-2 comparative evaluation: Electricity pricing, 1999.Google Scholar
Recommendations
On Streaming Consistency of Big Data Stream Processing in Heterogenous Clutsers
NBIS '15: Proceedings of the 2015 18th International Conference on Network-Based Information SystemsThere is currently a growing interest on Big Data Stream processing. With the increasing capabilities of Internet-based computing systems to generate, store and process Big Data Streams, various applications are benefiting from the information extracted ...
A Software Chain Approach to Big Data Stream Processing and Analytics
CISIS '15: Proceedings of the 2015 Ninth International Conference on Complex, Intelligent, and Software Intensive SystemsBig Data Stream processing is among the most important computing trends nowadays. The growing interest on Big Data Stream processing comes from the need of many Internet-based applications that generate huge data streams, whose processing can serve to ...
Two-stage scheduling for a fluctuant big data stream on heterogeneous servers with multicores in a data center
AbstractRapid processing with low-latency and high-throughput is a critical requirement for the applications of big data streams. However, the interferences among stream processing tasks in a data center decrease the utilization of the computational ...
Comments