Abstract
Writing enterprise grade software for multi-processor systems is an interesting challenge since such a system primarily involves a multitude of hardware components that exhibit conflict due to simultaneous access by unorganized software threads of user applications. The problem is particularly compounded with In-Memory paradigm that includes potential applications like Data Management in the modern era. With an emergence of distributed hardware trends like Non-Uniform Memory Access (NUMA), where access times to a system’s physical address space depend on relative location of Memory w.r.t CPU, it is crucial to rethink about placement of a user process’ workable memory with respect to executing threads. We present a few novel techniques from our Heap management work with SAP HANA as part of our goal towards building a strong NUMA awareness with in-memory databases. Our work primarily focuses on providing a robust and well-performant Memory Management framework on Linux OS by handling the associated complexity and challenges seen with enabling enterprise software to live on a distributed memory landscape. One of the important outcomes of our approach is to build a rich set of kernel APIs that provide fine-granular control to higher DBMS layers like Store and Query for educated placements of their relational data structures. However the generality of our techniques allows them to be readily applied to other domains that need to deal with NUMA performance penalty.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Kleen, A.: A NUMA API for linux (Technical whitepaper by SUSE Labs), April 2005. http://www.novell.com/docrep/2009/11/4621437_en.pdf
Lameter, C.: NUMA(non-uniform memory access): an overview. ACM Queue 11(7), 40 (2013)
Lameter, C.: Local and remote memory: memory in a Linux/NUMA System (June 2006)
Blagodurov, S., Zhuravlev, S., Fedorova, A., and Kamali, A.: A case for NUMA-aware contention management on multicore systems. In: Proceedings of PACT (2010)
Bryant, R., Barnes, J., Hawkes, J., Higdon, J., and Steiner, J.: Scaling Linux to the extreme. In: Proceedings of the 2004 Ottawa Linux Symposium, July 2004
Bueso, D., Norton, S.: An overview of kernel lock improvements. In: LinuxCon North America, Hewlett-Packard, Chicago, August 2014. http://events.linuxfoundation.org/sites/events/files/slides/linuxcon-2014-locking-final.pdf
Tam, D., Azimi, R., and Stumm, M.: Thread clustering: sharing-aware scheduling on SMP-CMP-SMT multiprocessors. In: ACM SIGOPS Operating Systems Review (2007)
Dashti, M., Fedorova, A., Funston, J., Gaud, F., et al.: Traffic management: a holistic approach to memory placement on NUMA systems. In: Proceedings of ASPLOS (2013)
Riel, R., Chegu, V.: Automatic NUMA balancing. red hat summit, April 2014. http://events.linuxfoundation.org/sites/events/files/slides/summit2014_riel_chegu_w_0340_automatic_numa_balancing_0.pdf
Schermerhorn, L.: Automatic page migration for Linux (2007). http://lca2007.linux.org.au/talk/197.html
Lang, H., Leis, V., Albutiu, M.-C., Neumann, T., Kemper, A.: Massively parallel NUMA-aware Hash joins. In: Jagatheesan, A., Levandoski, J., Neumann, T., Pavlo, A. (eds.) IMDM 2013/2014. LNCS, vol. 8921, pp. 3–14. Springer, Heidelberg (2015)
Albutiu, M., Kemper, A., Neumann, T.: Massively parallel sort-merge joins in main memory multi-core database systems. PVLDB 5(10), 1064–1075 (2012)
Li, Y., Pandis, I., Müller, R., Raman, V., Lohman, G.: NUMA-aware algorithms: the case of data shuffling. CIDR (2013)
Leis, V., Boncz, P., Kemper, A., Neumann, T.: Morsel-driven parallelism: a NUMA-aware query evaluation framework for Many-Core age. In: SIGMOD (2014)
Kaminski, P.: NUMA-aware heap memory manager
Ghemawat, S., Menage, P.: TCMalloc: thread-caching malloc. http://goog-perftools.sourceforge.net/doc/tcmalloc.html
Drepper, U.: What every programmer should know about memory, November 2007
Wagle, M., Booss, D., Schreter, I.: US patents with SAP Ref. No: 150150US01 (Title: NUMA-Aware Memory Allocation) and SAP Ref. No: 150423US01 (Title: NON-Uniform Memory Access (NUMA) database management system) (2015)
Gloger, W.: PTMalloc. http://www.malloc.de/en/
Berger, E., McKinley, K., Blumofe, R., Wilson, P.: Hoard: a scalable memory allocator for multithreaded applications. In: Proceedings of ASPLOS (2000)
Psaroudakis, I., Scheuer, T., May, N., Ailamaki, A.: Task scheduling for highly concurrent analytical and transactional main-memory workloads. ADMS@VLDB (2013)
Färber, F., May, N., Lehner, W., Große, P., Muller, I., Rauhe, H., Dees, J.: The SAP HANA database–an architecture overview. IEEE Data Eng. Bull. 35, 28–33 (2012)
libnuma–3rd party library for NUMA Policy control. http://linux.die.net/man/3/numa
Gaud, F., Lepers, B., Decouchant, J., Funston, J., Fedorova, A., Quéma, V.: Large pages may be harmful on NUMA systems. In: Proceedings of USENIX ATC (2014)
Ferreira, T.B., Matias, R., Macedo, A., Araujo, L.B.: An experimental study on memory allocators in multicore and multithreaded applications. In: Proceedings of PDCAT (2011)
Sentosa, M.: When Linux runs out of memory (2006). http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
Acknowledgements
The authors would like to thank Norman May, Tobias Scheuer, Iraklis Psaroudakis, Abdelkader Selami, Vipin Vishvkarma, Harshada Khandekar, Amit Saonerkar and the entire project team for their valuable contributions. The authors also thank Alexander Boehm, Werner Thesing and senior leadership for their valuable guidance.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Wagle, M., Booss, D., Schreter, I., Egenolf, D. (2016). NUMA-Aware Memory Management with In-Memory Databases. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking: Traditional to Big Data to Internet of Things. TPCTC 2015. Lecture Notes in Computer Science(), vol 9508. Springer, Cham. https://doi.org/10.1007/978-3-319-31409-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-31409-9_4
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-31408-2
Online ISBN: 978-3-319-31409-9
eBook Packages: Computer ScienceComputer Science (R0)