Abstract
Anomalies in water quality, which frequently arise due to pollution, constitute a substantial menace to human health. The preservation of public welfare critically entails the timely recognition of abnormal water quality. Conventional techniques for detecting water quality anomalies face obstacles such as the necessity of expert knowledge, limited accuracy in detection, and delays in identification. In this paper, we proposed an original unsupervised technique for identifying water quality anomalies combined with time-frequency analysis and clustering (TCAD). We chose time-frequency analysis because it effectively evaluates water quality changes, generating distinct multi-band signals that reflect different aspects of water quality dynamics. We also proposed a clustering technique which can identify water quality markers and amalgamate data from multi-band signals for accurate anomaly detection. We seek to clarify the reasoning behind our methodology by portraying how time-frequency analysis and clustering address the deficiencies of conventional methods. Our experiments evaluated various indicators of water quality, and the effectiveness of our proposed approach was supported by comparative analyses with commonly used models for detecting anomalies in water quality.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and materials
All data generated or analyzed during this study are included in this published article.
References
Akbari Z, Unland R (2016) Automated determination of the input parameter of DBSCAN based on outlier detection. In: Artificial Intelligence Applications and Innovations: 12th IFIP WG 12.5 International Conference and Workshops, AIAI 2016, Thessaloniki, Greece, September 16-18, 2016, Proceedings 12, Springer, pp 280–291
An P, Wang Z, Zhang C (2022) Ensemble unsupervised autoencoders and gaussian mixture model for cyberattack detection. Inf Process Manag 59(2):102844
Belhadi A, Djenouri Y, Srivastava G et al (2020) A two-phase anomaly detection model for secure intelligent transportation ride-hailing trajectories. IEEE Trans Intell Transp Syst 22(7):4496–4506
Dogo EM, Nwulu NI, Twala B et al (2019) A survey of machine learning methods applied to anomaly detection on drinking-water quality data. Urban Water J 16(3):235–248
Erfani SM, Rajasegarar S, Karunasekera S et al (2016) High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning. Pattern Recognit 58:121–134
Foroumandi E, Nourani V, Dabrowska D et al (2022) Linking spatial-temporal changes of vegetation cover with hydroclimatological variables in terrestrial environments with a focus on the lake Urmia basin. Land 11(1):115
Foroumandi E, Nourani V, Kantoush SA (2022) Investigating the main reasons for the tragedy of large saline lakes: drought, climate change, or anthropogenic activities? A call to action. Journal of Arid Environ 196(104):652
Foroumandi E, Nourani V, Huang JJ et al (2023) Drought monitoring by downscaling grace-derived terrestrial water storage anomalies: a deep learning approach. Journal of Hydrology 616
He HM, Hou DB, Zhao HF et al (2013) Multi-parameters fusion algorithm for detecting anomalous water quality. J Zhejiang Univ Eng Sci 47(4):735–740
Hilal W, Gadsden SA, Yawney J (2021) A review of anomaly detection techniques and applications in financial fraud. Expert Syst Appl 116429
Huang P, Wang L, Hou D et al (2021) A feature extraction method based on the entropy-minimal description length principle and GBDT for common surface water pollution identification. J Hydroinformatics 23(5):1050–1065
Jalal D, Ezzedine T (2020) Decision tree and support vector machine for anomaly detection in water distribution networks. In: 2020 International Wireless Communications and Mobile Computing (IWCMC), IEEE, pp 1320–1323
Jiang X, Song Q, Du G et al (2023) A review on research and application of variational mode decomposition. J Instrum
Lesouple J, Baudoin C, Spigai M et al (2021) Generalized isolation forest for anomaly detection. Pattern Recognit Lett 149:109–119
Liu J, Wang P, Jiang D et al (2020) An integrated data-driven framework for surface water quality anomaly detection and early warning. J Clean Prod 251:119145
Lung WS (2022) River bod/do modeling at new normal. In: Water Quality Modeling That Works. Springer, p 69–105
McKenna SA, Klise KA, Wilson MP (2008) Testing water quality change detection algorithms. Water Distribution Systems Analysis Symposium 2006:1–15
Mendivil-García K, Amabilis-Sosa LE, Salinas-Juárez MG et al (2022) Climate change impact assessment on a tropical river resilience using the Streeter-Phelps dissolved oxygen model. Front Environ Sci 10
Muharemi F, Logofătu D, Leon F (2019) Machine learning approaches for anomaly detection of water quality on a real-world data set. J Inf Telecommun 3(3):294–307
Narayan A, Madhu Kumar S, Chacko AM (2023) A review of financial fraud detection in e-commerce using machine learning. In: Intelligent Data Engineering and Analytics: Proceedings of the 10th International Conference on Frontiers in Intelligent Computing: Theory and Applications (FICTA 2022), Springer, pp 237–248
Ni Q, Cao X, Tan C et al (2022) An improved graph convolutional network with feature and temporal attention for multivariate water quality prediction. Environ Sci Pollut Res 1–14
Nourani V, Ghaneei P, Kantoush SA (2022) Robust clustering for assessing the spatiotemporal variability of groundwater quantity and quality. J Hydrol 604:127272
Oliker N, Ostfeld A (2014) Comparison of multivariate classification methods for contamination event detection in water distribution systems. Procedia Eng 70:1271–1279
Pang G, Shen C, Cao L et al (2021) Deep learning for anomaly detection: a review. ACM computing surveys (CSUR) 54(2):1–38
Peterson KT, Sagan V, Sloan JJ (2020) Deep learning-based water quality estimation and anomaly detection using Landsat-8/Sentinel-2 virtual constellation and cloud computing. GIScience Remote Sens 57(4):510–525
Qin w, Fu y (2018) Water quality abnormity inspection and detection via the vector auto-regressive model. J Saf Environ 18(04):1560–1563
Raciti M, Cucurull J, Nadjm-Tehrani S (2012) Anomaly detection in water management systems. In: Critical infrastructure protection. Springer, p 98–119
Rodriguez-Perez J, Leigh C, Liquet B et al (2020) Detecting technical anomalies in high-frequency water-quality data using artificial neural networks. Environ Sci Technol 54(21):13719–13730
Shaukat K, Alam TM, Luo S et al (2021) A review of time-series anomaly detection techniques: a step to future perspectives. In: Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 1, Springer, pp 865–877
Zhang J, Zhu X, Yue Y et al (2017) A real-time anomaly detection algorithm/or water quality data using dual time-moving windows. In: 2017 Seventh international conference on innovative computing technology (INTECH), IEEE, pp 36–41
Zhi X, Jing M, Hao W et al (2019) Key indicator and critical condition for the water resource carrying capacity in the Yangtze river estuary. Journal of Tsinghua University (Science and Technology) 59(5):364–372
Zhong S, Zhang K, Bagheri M et al (2021) Machine learning: new ideas and tools in environmental science and engineering. Environ Sci Technol 55(19):12741–12754
Funding
This paper is supported by the National Natural Science Foundation of China (12273003).
Author information
Authors and Affiliations
Contributions
The study conception and design, data collection and analysis, guide of experiments, and paper revision were performed by Qingjian Ni. The design of model, code implementation, experiments, the first draft of the manuscript, and paper revision were performed by Xuehan Cao. Jiayi Yuan and Chaoqun Tan commented on previous versions of the manuscript and made suggestions for revision. Ziqi Zhao was responsible for creating specific diagrams and making various revisions to this paper.
Corresponding author
Ethics declarations
Ethical approval and consent to participate
All authors have read and agreed to the published version of the manuscript.
Consent for publication
All authors have read and agreed to the published version of the manuscript.
Conflict of interest
The authors declare no competing interests.
Additional information
Responsible editor: Xianliang Yi.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ni, Q., Cao, X., Zhao, Z. et al. An unsupervised water quality anomaly detection method based on a combination of time-frequency analysis and clustering. Environ Sci Pollut Res 31, 15920–15931 (2024). https://doi.org/10.1007/s11356-024-32170-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11356-024-32170-y