Skip to main content
SpringerLink
Log in

ATUN-HL: Auto Tuning of Hybrid Layouts Using Workload and Data Characteristics

  • Conference paper
  • First Online:
Book cover Advances in Databases and Information Systems (ADBIS 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11019))

Included in the following conference series:

Abstract

Ad-hoc analysis implies processing data in near real-time. Thus, raw data (i.e., neither normalized nor transformed) is typically dumped into a distributed engine, where it is generally stored into a hybrid layout. Hybrid layouts divide data into horizontal partitions and inside each partition, data are stored vertically. They keep statistics for each horizontal partition and also support encoding (i.e., dictionary) and compression to reduce the size of the data. Their built-in support for many ad-hoc operations (i.e., selection, projection, aggregation, etc.) makes hybrid layouts the best choice for most operations.

Horizontal partition and dictionary sizes of hybrid layouts are configurable and can directly impact the performance of analytical queries. Hence, their default configuration cannot be expected to be optimal for all scenarios. In this paper, we present ATUN-HL (Auto TUNing Hybrid Layouts), which based on a cost model and given the workload and the characteristics of data, finds the best values for these parameters. We prototyped ATUN-HL for Apache Parquet, which is an open source implementation of hybrid layouts in Hadoop Distributed File System, to show its effectiveness. Our experimental evaluation shows that ATUN-HL provides on average 85% of all the potential performance improvement, and 1.2x average speedup against default configuration.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://hadoop.apache.org.

  2. 2.

    https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html.

  3. 3.

    https://spark.apache.org.

  4. 4.

    https://orc.apache.org.

  5. 5.

    https://parquet.apache.org.

  6. 6.

    https://carbondata.apache.org.

  7. 7.

    http://www.tpc.org/tpch.

  8. 8.

    https://math.stackexchange.com/questions/72223/finding-expected-number-of-distinct-values-selected-from-a-set-of-integers.

  9. 9.

    http://www.ac.upc.edu/serveis-tic/altas-prestaciones.

  10. 10.

    http://www.tpc.org/tpcds.

References

  1. Abedjan, Z., Golab, L., Naumann, F.L.: Data profiling: a tutorial. In: SIGMOD Conference. ACM (2017)

    Google Scholar 

  2. Alagiannis, I., Idreos, S., Ailamaki, A.: H2O: a hands-free adaptive store. In: SIGMOD Conference. ACM (2014)

    Google Scholar 

  3. Azim, T., Karpathiotakis, M., Ailamaki, A.: Recache: reactive caching for fast analytics over heterogeneous data. PVLDB 11(3), 324–337 (2017)

    Google Scholar 

  4. Bian, H., et al.: Wide table layout optimization based on column ordering and duplication. In: SIGMOD Conference. ACM (2017)

    Google Scholar 

  5. Cardenas, A.F.: Analysis and performance of inverted data base structures. Commun. ACM 18(5), 253–263 (1975)

    Article  MathSciNet  Google Scholar 

  6. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  7. Ferreira, M., Paiva, J., Bravo, M., Rodrigues, L.E.T.: Smartfetch: efficient support for selective queries. In: CloudCom. IEEE Computer Society (2015)

    Google Scholar 

  8. Han, J., Cheng, H., Xin, D., Yan, X.: Frequent pattern mining: current status and future directions. Data Min. Knowl. Discov. 15(1), 55–86 (2007)

    Article  MathSciNet  Google Scholar 

  9. Herodotou, H., Babu, S.: Profiling, what-if analysis, and cost-based optimization of mapreduce programs. PVLDB 4(11), 1111–1122 (2011)

    Google Scholar 

  10. Herodotou, H., et al.: Starfish: a self-tuning system for big data analytics. In: CIDR (2011)

    Google Scholar 

  11. Jindal, A., Quiané-Ruiz, J., Dittrich, J.: Trojan data layouts: right shoes for a running elephant. In: SoCC. ACM (2011)

    Google Scholar 

  12. Li, Y., Patel, J.M.: Widetable: an accelerator for analytical data processing. PVLDB 7(10), 907–918 (2014)

    Google Scholar 

  13. Moerkotte, G.: Small materialized aggregates: a light weight index structure for data warehousing. In: VLDB, pp. 476–487 (1998)

    Google Scholar 

  14. Munir, R.F., Abelló, A., Romero, O., Thiele, M., Lehner, W.: A cost-based storage format selector for materialization in big data frameworks. CoRR, abs/1806.03901 (2018)

    Google Scholar 

  15. Munir, R.F., Romero, O., Abelló, A., Bilalli, B., Thiele, M., Lehner, W.: ResilientStore: a heuristic-based data format selector for intermediate results. In: Bellatreche, L., Pastor, Ó., Almendros Jiménez, J.M., Aït-Ameur, Y. (eds.) MEDI 2016. LNCS, vol. 9893, pp. 42–56. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45547-1_4

    Chapter  Google Scholar 

  16. Shvachko, K.V.: HDFS scalability: the limits to growth. Login 35(2), 6–16 (2010)

    Google Scholar 

  17. Sun, L., Franklin, M.J., Krishnan, S., Xin, R.S.: Fine-grained partitioning for aggressive data skipping. In: SIGMOD Conference. ACM (2014)

    Google Scholar 

  18. Sun, L., Franklin, M.J., Wang, J., Wu, E.: Skipping-oriented partitioning for columnar layouts. PVLDB 10(4), 421–432 (2016)

    Google Scholar 

Download references

Acknowledgement

This research has been funded by the European Commission through the Erasmus Mundus Joint Doctorate “Information Technologies for Business Intelligence - Doctoral College” (IT4BI-DC), and the GENESIS project, funded by the Spanish Ministerio de Ciencia e Innovación under project TIN2016-79269-R.

Author information

Authors and Affiliations

  1. Universitat Politècnica de Catalunya (UPC), Barcelona, Spain

    Rana Faisal Munir, Alberto Abelló & Oscar Romero

  2. Technische Universität Dresden (TUD), Dresden, Germany

    Rana Faisal Munir, Maik Thiele & Wolfgang Lehner

Authors
  1. Rana Faisal Munir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alberto Abelló
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oscar Romero
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maik Thiele
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wolfgang Lehner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rana Faisal Munir .

Editor information

Editors and Affiliations

  1. Eötvös Loránd University, Budapest, Hungary

    András Benczúr

  2. Christian-Albrechts-Universität, Kiel, Germany

    Bernhard Thalheim

  3. Eötvös Loránd University, Budapest, Hungary

    Tomáš Horváth

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Munir, R.F., Abelló, A., Romero, O., Thiele, M., Lehner, W. (2018). ATUN-HL: Auto Tuning of Hybrid Layouts Using Workload and Data Characteristics. In: Benczúr, A., Thalheim, B., Horváth, T. (eds) Advances in Databases and Information Systems. ADBIS 2018. Lecture Notes in Computer Science(), vol 11019. Springer, Cham. https://doi.org/10.1007/978-3-319-98398-1_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-98398-1_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-98397-4

  • Online ISBN: 978-3-319-98398-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation