Abstract
One of the main development resources for website engineers are Web templates. Templates allow them to increase productivity by plugin content into already formatted and prepared pagelets. For the final user templates are also useful, because they provide uniformity and a common look and feel for all webpages. However, from the point of view of crawlers and indexers, templates are an important problem, because templates usually contain irrelevant information such as advertisements, menus, and banners. Processing and storing this information leads to a waste of resources (storage space, bandwidth, etc.). It has been measured that templates represent between 40 % and 50 % of data on the Web. Therefore, identifying templates is essential for indexing tasks. In this work we propose a novel method for automatic web template extraction that is based on similarity analysis between the DOM trees of a collection of webpages that are detected using an hyperlink analysis. Our implementation and experiments demonstrate the usefulness of the technique.
This work has been partially supported by the EU (FEDER) and the Spanish Ministerio de Economía y Competitividad (Secretaría de Estado de Investigación, Desarrollo e Innovación) under grant TIN2013-44742-C4-1-R and by the Generalitat Valenciana under grant PROMETEOII/2015/013. David Insa was partially supported by the Spanish Ministerio de Eduación under FPU grant AP2010-4415.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Some authors answered that their benchmarks were not stored for future use, or that they did not save the gold standard.
References
Alarte, J., Insa, D., Silva, J., Tamarit, S.: Automatic detection of webpages that share the same web template. In: ter Beek, M.H., Ravara, A. (eds.) Proceedings of the 10th International Workshop on Automated Specification and Verification of Web Systems (WWV 2014). Electronic Proceedings in Theoretical Computer Science, vol. 163, pp. 2–15. Open Publishing Association, July 2014
Alarte, J., Insa, D., Silva, J., Tamarit, S.: Web template extraction based on hyperlink analysis. In: Escobar, S. (ed.) Proceedings of the XIV Jornadas sobre Programación y Lenguajes (PROLE 2015). Electronic Proceedings in Theoretical Computer Science, vol. 173, pp. 16–26. Open Publishing Association, September 2015
Bar-Yossef, Z., Rajagopalan, S.: Template detection via data mining and its applications. In: Proceedings of the 11th International Conference on World Wide Web (WWW 2002), pp. 580–591. ACM, New York (2002)
Baroni, M., Chantree, F., Kilgarriff, A., Sharoff, S.: Cleaneval: a competition for cleaning web pages. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2008), pp. 638–643. European Language Resources Association, May 2008
Burget, R., Rudolfova, I.: Web page element classification based on visual features. In: Proceedings of the 1st Asian Conference on Intelligent Information and Database Systems (ACIIDS 2009), pp. 67–72. IEEE Computer Society, Washington, DC (2009)
Cardoso, E., Jabour, I., Laber, E., Rodrigues, R., Cardoso, P.: An efficient language-independent method to extract content from news webpages. In: Proceedings of the 11th ACM Symposium on Document Engineering (DocEng 2011), pp. 121–128. ACM, New York (2011)
Chakrabarti, S.: Integrating the Document Object Model with hyperlinks for enhanced topic distillation and information extraction. In: Proceedings of the 10th International Conference on World Wide Web (WWW 2001), pp. 211–220. ACM, New York (2001)
W3C Consortium. Document Object Model (DOM) (1997). http://www.w3.org/DOM/
Ferraresi, A., Zanchetta, E., Baroni, M., Bernardini, S.: Introducing and evaluating ukWaC, a very large web-derived corpus of English. In: Proceedings of the 4th Web as Corpus Workshop (WAC-4), pp. 47–54 (2008)
Gibson, D., Punera, K., Tomkins, A.: The volume and evolution of web page templates. In: Ellis, A., Hagino, T. (eds.) Proceedings of the 14th International Conference on World Wide Web (WWW 2005), pp. 830–839. ACM, May 2005
Gottron, T.: Content code blurring: a new approach to content extraction. In: Tjoa, A.M., Wagner, R.R. (eds.) Proceedings of the 19th International Workshop on Database and Expert Systems Applications (DEXA 2008), pp. 29–33. IEEE Computer Society, September 2008
Insa, D., Silva, J., Tamarit, S.: Using the words/leafs ratio in the DOM tree for content extraction. J. Logic Algebraic Program. 82(8), 311–325 (2013)
Kadam, V., Devale, P.R.: A methodology for template extraction from heterogeneous web pages. Indian J. Comput. Sci. Eng. (IJCSE) 3(3), 449–452 (2012)
Kohlschütter, C.: A densitometric analysis of web template content. In: Quemada, J., León, G., Maarek, Y.S., Nejdl, W. (eds.) Proceedings of the 18th International Conference on World Wide Web (WWW 2009), pp. 1165–1166. ACM, April 2009
Kohlschütter, C., Fankhauser, P., Nejdl, W.: Boilerplate detection using shallow text features. In: Davison, B.D., Suel, T., Craswell, N., Liu, B. (eds.) Proceedings of the 3th International Conference on Web Search and Web Data Mining (WSDM 2010), pp. 441–450. ACM, February 2010
Kohlschütter, C., Nejdl, W.: A densitometric approach to web page segmentation. In: Shanahan, J.G., Amer-Yahia, S., Manolescu, I., Zhang, Y., Evans, D.A., Kolcz, A., Choi, K.-S., Chowdhury, A. (eds.) Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM 2008), pp. 1173–1182. ACM, October 2008
Nguyen, D.Q., Nguyen, D.Q., Pham, S.B., Bui, T.D.: A fast template-based approach to automatically identify primary text content of a web page. In: Proceedings of the 2009 International Conference on Knowledge and Systems Engineering, KSE 2009, pp. 232–236. IEEE Computer Society (2009)
de Castro Reis, D., Golgher, P.B., Silva, A.S., Laender, A.H.F.: Automatic web news extraction using tree edit distance. In: Proceedings of the 13th International Conference on World Wide Web (WWW 2004), pp. 502–511. ACM, New York (2004)
Vieira, K., da Silva, A.S., Pinto, N., de Moura, E.S., Cavalcanti, J.M.B., Freire, J.: A fast and robust method for web page template detection and removal. In: Proceedings of the 15th ACM International Conference on Information and Knowledge Management (CIKM 2006), pp. 258–267. ACM, New York (2006)
Weninger, T., Hsu, W.H., Han, J.: CETR: content extraction via tag ratios. In: Rappa, M., Jones, P., Freire, J., Chakrabarti, S. (eds.) Proceedings of the 19th International Conference on World Wide Web (WWW 2010), pp. 971–980. ACM, April 2010
Yi, L., Liu, B., Li, X.: Eliminating noisy information in web pages for data mining. In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2003), pp. 296–305. ACM, New York (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Alarte, J., Insa, D., Silva, J., Tamarit, S. (2016). Site-Level Web Template Extraction Based on DOM Analysis. In: Mazzara, M., Voronkov, A. (eds) Perspectives of System Informatics. PSI 2015. Lecture Notes in Computer Science(), vol 9609. Springer, Cham. https://doi.org/10.1007/978-3-319-41579-6_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-41579-6_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-41578-9
Online ISBN: 978-3-319-41579-6
eBook Packages: Computer ScienceComputer Science (R0)