Abstract
We describe a compression model for semistructured documents, called Structural Contexts Model, which takes advantage of the context information usually implicit in the structure of the text. The idea is to use a separate semiadaptive model to compress the text that lies inside each different structure type (e.g., different XML tag). The intuition behind the idea is that the distribution of all the texts that belong to a given structure type should be similar, and different from that of other structure types. We test our idea using a word-based Huffman coding, which is the standard for compressing large natural language textual databases, and show that our compression method obtains significant improvements in compression ratios. We also analyze the possibility that storing separate models may not pay off if the distribution of different structure types is not different enough, and present a heuristic to merge models with the aim of minimizing the total size of the compressed database. This technique gives an additional improvement over the plain technique. The comparison against existing prototypes shows that our method is a competitive choice for compressed text databases. Finally, we show how to apply SCM over text chunks, which allows one to adjust the different word frequencies as they change across the text collection.
This work was partially supported by CYTED VII.19 RIBIDI project (all authors) and Fondecyt Project 1-020831 (second author).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Buchsbaum, A.L., Caldwell, D.F., Ward Church, K., Fowler, G.S., Muthukrishnan, S.: Engineering the compression of massive tables: an experimental approach. In: Symposium on Discrete Algorithms, pp. 175–184 (2000)
Bentley, J., Sleator, D., Tarjan, R., Wei, V.: A locally adaptive data compression scheme. Communications of the ACM 29, 320–330 (1986)
Cheney, J.: Compressing XML with multiplexed hierarchical PPM models. In: Proc. Data Compression Conference (DCC 2001), p. 163 (2001)
Dvorský, J., Pokorný, J., Snásel, V.: Word-based compression methods and indexing for text retrieval systems. In: Eder, J., Rozman, I., Welzer, T. (eds.) ADBIS 1999. LNCS, vol. 1691, pp. 75–84. Springer, Heidelberg (1999)
Harman, D.: Overview of the Third Text REtrieval Conference. In: Proc. Third Text REtrieval Conference (TREC-3), pp. 1–19. NIST Special Publication 500-207 (1995)
Heaps, H.S.: Information Retrieval - Computational and Theoretical Aspects. Academic Press, London (1978)
Huffman, D.A.: A method for the construction of minimum-redundancy codes. Proc. Inst. Radio Engineers 40(9), 1098–1101 (1952)
Buchsbaum, A.L., Caldwell, D.F., Ward Church, K., Fowler, G.S., Muthukrishnan, S.: Engineering the compression of massive tables: an experimental approach. In: Symposium on Discrete Algorithms, pp. 175–184 (2000)
Silva de Moura, E., Navarro, G., Ziviani, N., Baeza-Yates, R.: Fast and flexible word searching on compressed text. ACM Transactions on Information Systems 18(2), 113–139 (2000)
Moffat, A.: Word-based text compression. Software - Practice and Experience 19(2), 185–198 (1989)
Moffat, A., Wan, R.: RE-store: A system for compressing, browsing and searching large documents. In: Proc. 8th Intl. Symp. on String Processing and Information Retrieval (SPIRE 2001), pp. 162–174 (2001)
Navarro, G., Silva de Moura, E., Neubert, M., Ziviani, N., Baeza-Yates, R.: Adding compression to block addressing inverted indexes. Information Retrieval 3(1), 49–77 (2000)
Shannon, C.: A mathematical theory of communication. Bell Syst. Tech. J. 27, 398–403 (1948)
Witten, I.H., Bell, T.C., Cleary, J.G.: Text Compression. Prentice Hall, Englewood Cliffs (1990)
Tolani, P., Haritsa, J.R.: XGRIND: A query-friendly XML compressor. In: ICDE (2002), http://citeseer.nj.nec.com/503319.html
Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes, 2nd edn. Morgan Kaufmann Publishers, Inc., San Francisco (1999)
Ziv, J., Lempel, A.: An universal algorithm for sequential data compression. IEEE Trans. on Information Theory 23(3), 337–343 (1977)
Ziviani, N., Moura, E., Navarro, G., Baeza-Yates, R.: Compression: A key for next-generation text retrieval systems. IEEE Computer 33(11), 37–44 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Adiego, J., Navarro, G., de la Fuente, P. (2003). SCM: Structural Contexts Model for Improving Compression in Semistructured Text Databases. In: Nascimento, M.A., de Moura, E.S., Oliveira, A.L. (eds) String Processing and Information Retrieval. SPIRE 2003. Lecture Notes in Computer Science, vol 2857. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-39984-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-540-39984-1_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20177-9
Online ISBN: 978-3-540-39984-1
eBook Packages: Springer Book Archive