Abstract
Knowledge Discovery in Databases (KDD) is an iterative process that aims at extracting interesting, previously unknown and hidden patterns from huge databases. Use of objective measures of interestingness in popular data mining algorithms often leads to another data mining problem, although of reduced complexity. The reduction in the volume of the discovered rules is desirable in order to improve the efficiency of the overall KDD process. Subjective measures of interestingness are required to achieve this. In this paper we study novelty of the discovered rules as a subjective measure of interestingness. We propose a framework to quantify novelty of the discovered rules in terms of their deviations from the known rules. The computations are carried out using the importance that the user gives to different deviations. The computed degree of novelty is then compared with the user given threshold to report novel rules to the user. We implement the proposed framework and experiment with some public datasets. The experimental results are quite promising.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P.: From Data Mining to Knowledge Discovery: An Overview. In: Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R. (eds.) Advances in Knowledge Discovery and Data Mining, AAAI/MIT Press, Menlo Park, CA (1996)
Pyle, D.: Data Preparation for Data Mining. Morgan Kaufmann, San Francisco (1999)
Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. John Wiley & Sons (Asia) PV. Ltd., Chichester (2002)
Liu, B., Hsu, W., Chen, S.: Using General Impressions to Analyze Discovered Classification Rules. In: Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, KDD 1997 (1997)
Piateskey-Shapiro, G., Matheus, C.J.: The Interestingness of Deviations. In: Proceedings of AAAI Workshop on Knowledge Discovery in Databases (1994)
Liu, B., Hsu, W., Mun, L., Lee, H.: Finding Interesting Patterns Using User Expectations. Technical Report:TRA7/96. Department of Information Systems and Computer Science, National University of Singapore (1996)
Silberschatz, A., Tuzhilin, A.: On Subjective Measures of Interestingness in Knowledge Discovery. In: Proceedings of the 1st International Conference on Knowledge Discovery and Data Mining (1995)
Silberschatz, A., Tuzhilin, A.: What Makes Patterns Interesting in Knowledge Discovery Systems. IEEE Transactions on Knowledge and Data Engineering 5(6) (1996)
Liu, B., Hsu, W.: Post Analysis of Learned Rules. In: Proceedings of the 13th National Conference on AI(AAAI 1996) (1996)
Padmanabhan, B., Tuzhilin, A.: Unexpectedness as a Measure of Interestingness in Knowledge Discovery. Working paper # IS-97-6, Dept. of Information Systems, Stern School of Business, NYU (1997)
Kohonen, T.: Self-Organization and Associative Memory, 3rd edn. Springer, Berlin (1993)
Yairi, T., Kato, Y., Hori, K.: Fault Detection by Mining Association Rules from House-keeping Data. In: Proceedings of International Symposium on Artificial Intelligence, Robotics and Automation in Space, SAIRAS 2001 (2001)
Marsland, S.: On-Line Novelty Detection Through Self-Organization, with Application to Robotics. Ph.D. Thesis, Department of Computer Science, University of Manchester (2001)
Basu, S., Mooney, R.J., Pasupuleti, K.V., Ghosh, J.: Using Lexical Knowledge to Evaluate the Novelty of Rules Mined from Text. In: Proceedings of the NAACL workshop and other Lexical Resources: Applications, Extensions and Customizations (2001)
Liu, B., Hsu, W., Ma, Y.: Integrating Classification and Association Rule Mining. In: Proceedings of the 4th International Conference on Knowledge Discovery and Data Mining, KDD 1998 (1998)
Pujari, A.K.: Data Mining Techniques. 1st edn. Universities Press(India) Limited (2001)
Dunham, M.H.: Data Mining: Introductory and Advanced Topics, 1st edn. Pearson Education (Singaphore) Pte. Ltd., London (2003)
Williams, G.J.: Evolutionary Hot Spots Data Mining: An Architecture for Exploring for Interesting Discoveries. In: Zhong, N., Zhou, L. (eds.) PAKDD 1999. LNCS (LNAI), vol. 1574, pp. 184–193. Springer, Heidelberg (1999)
Psaila, G.: Discovery of Association Rule Meta-Patterns. In: Mohania, M., Tjoa, A.M. (eds.) DaWaK 1999. LNCS, vol. 1676, pp. 219–228. Springer, Heidelberg (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Al-Hegami, A.S., Bhatnagar, V., Kumar, N. (2004). Novelty Framework for Knowledge Discovery in Databases. In: Kambayashi, Y., Mohania, M., Wöß, W. (eds) Data Warehousing and Knowledge Discovery. DaWaK 2004. Lecture Notes in Computer Science, vol 3181. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30076-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-540-30076-2_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22937-7
Online ISBN: 978-3-540-30076-2
eBook Packages: Springer Book Archive