Abstract
MapReduce is an effective tool for processing large amounts of data in parallel using a cluster of processors or computers. One common data processing task is the join operation, which combines two or more datasets based on values common to each. In this paper, we present a network aware multi-way join for MapReduce(NAMM) that improves performance by redistributing the workload amongst reducers. NAMM achieves this by redistributing tuples directly between reducers with an intelligent network aware algorithm. We show that our presented technique has significant potential to minimize the time required to join multiple datasets.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51, 107–113 (2008)
Hoefler, T., Lumsdaine, A., Dongarra, J.: Towards Efficient MapReduce Using MPI. In: Ropo, M., Westerholm, J., Dongarra, J. (eds.) PVM/MPI. LNCS, vol. 5759, pp. 240–249. Springer, Heidelberg (2009)
White, T.: Hadoop the definitive guide, 2nd edn. O’Reilly, Sebastopol (2010)
Afrati, F.N., Ullman, J.D.: Optimizing Multiway Joins in a Map-Reduce Environment. IEEE Transactions on Knowledge and Data Engineering 23, 1282–1298 (2011)
Lynden, S., Tanimura, Y., Kojima, I., Matono, A.: Dynamic Data Redistribution for MapReduce Joins. In: 2011 IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), pp. 717–723 (2011)
Chandar, J.: Join Algorithms using Map/Reduce, Master of Science, School of Informatics, University of Edinburgh (2010)
Wang, X., Burns, R., Terzis, A., Deshpande, A.: Network-aware join processing in global-scale database federations. In: IEEE 24th International Conference on Data Engineering, ICDE 2008, pp. 586–595 (2008)
Hunt, P., Konar, M., Junqueira, F.P., Reed, B.: ZooKeeper: Wait-free coordination for Internet-scale systems. In: USENIX ATC (2010)
Palla, K.: A Comparative Analysis of Join Algorithms Using the Hadoop Map/Reduce Framework, Master of Science, School of Informatics, University of Edinburgh (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Slagter, K., Hsu, CH., Chung, YC., Park, J.H. (2013). Network-Aware Multiway Join for MapReduce. In: Park, J.J.(.H., Arabnia, H.R., Kim, C., Shi, W., Gil, JM. (eds) Grid and Pervasive Computing. GPC 2013. Lecture Notes in Computer Science, vol 7861. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38027-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-642-38027-3_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-38026-6
Online ISBN: 978-3-642-38027-3
eBook Packages: Computer ScienceComputer Science (R0)