Abstract
Dividing web pages into fragments has been shown to provide significant benefits for both content generation and caching. However, the lack of good methods to analyze interesting fragments in large collections of web pages is preventing existing large web sites from using fragment-based techniques. Fragments are considered to be interesting if they are completely or structurally shared among multiple web pages. This paper first gives a formal description of the problem, and then presents our system for shared fragments analysis. We propose a well-designed data structure for representing web pages, and develop an efficient algorithm by utilizing database techniques. Our system is unique in its shared fragments analysis for large collections of web pages. The system has been built and successfully applied to some sets of large web pages, which has shown its effectiveness and usefulness, and may serve as a core building block in many applications.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Challenger, J., et al.: A Publishing System for Efficiently Creating Dynamic Web Content. In: Proceedings of INFOCOM 2000 (March 2000)
BEA WebLogic Server. http://www.bea.com/products/weblogic/server/
Oracle9iAS, http://www.oracle.com/appserver/
Microsoft. Caching Architecture Guide for.NET Framework Applications (2003)
IBM WebSphere. http://www-3.ibm.com/software/webservers/appserv/
Datta, A., et al.: Proxy-Based Acceleration of Dynamically Generated Content on the World Wide Web: An approach and Implementation. In: Proceeding of ACM SIGMOD Intl. Conf. on Management of Data, June 2002, pp. 97–108 (2002)
Yuan, C., Hua, Z., Zhang, Z.: Proxy+: Simple Proxy Augmentation for Dynamic Content Processing. In: WCW 2003 (2003)
ESI Consortium. Edge Side Includes, http://www.esi.org
Document Object Model – W3C Recommendation, http://www.w3.org/DOM
Network Working Group.: Digest Values for DOM (DOMHASH). RFC 2803 (April 2000)
CyberNeko HTML Parser, http://people.apache.org/~andyc/neko/doc/index.html
Broder, A.Z.: Some Applications of Rabin’s Fingerprinting Method. In: Capocelli, R., De Santis, A., Vaccaro, U. (eds.) Sequences II: Methods in Communications, Security, and Computer Science, pp. 143–152. Springer, Heidelberg (1993)
Broder, A.Z.: On the Resemblance and Containment of Documents. In: Proceedings of SEQUENCES 1997 (1997)
Buttler, D., Liu, L.: A Fully Automated Object Extraction System for the World Wide Web. In: Proceedings of ICDCS 2001 (2001)
Kamiya, T., et al.: CCFinder: A Multilinguistic Token-Based Code Clone Detection System for Large Scale Source Code. In: IEEE Transactions on Software Engineering (July 2002)
Ramaswamy, L., lyengar, A., Liu, L., Douglis, F.: Automatic Detection of Fragments in Dynamically Generated Web Pages. In: WWW 2004, New York (May 2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ma, J., Gu, Z. (2006). A Shared Fragments Analysis System for Large Collections of Web Pages. In: Bunke, H., Spitz, A.L. (eds) Document Analysis Systems VII. DAS 2006. Lecture Notes in Computer Science, vol 3872. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11669487_35
Download citation
DOI: https://doi.org/10.1007/11669487_35
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-32140-8
Online ISBN: 978-3-540-32157-6
eBook Packages: Computer ScienceComputer Science (R0)