Abstract
Recent publications have emphasised map-reduce as a general programming model (labelled Google map-reduce), and described existing high-performance implementations for large data sets. We present two parallel implementations for this Google map-reduce skeleton, one following earlier work, and one optimised version, in the parallel Haskell extension Eden. Eden’s specific features, like lazy stream processing, dynamic reply channels, and nondeterministic stream merging, support the efficient implementation of the complex coordination structure of this skeleton. We compare the two implementations of the Google map-reduce skeleton in usage and performance, and deliver runtime analyses for example applications. Although very flexible, the Google map-reduce skeleton is often too general, and typical examples reveal a better runtime behaviour using alternative skeletons.
Chapter PDF
Similar content being viewed by others
Keywords
- Parallel Implementation
- Intermediate Data
- Collective Operation
- Algorithmic Skeleton
- Productive Implementation
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Cole, M.I.: Algorithmic Skeletons: Structured Management of Parallel Computation. Research Monographs in Parallel and Distributed Comp. MIT Press, Cambridge (1989)
Rabhi, F.A., Gorlatch, S. (eds.): Patterns and Skeletons for Parallel and Distributed Computing. Springer, Heidelberg (2003)
MPI-2: Extensions to the message-passing interface. Technical report, University of Tennessee, Knoxville (July 1997)
Gamma, E., Helm, R., Johnson, R.E., Vlissides, J.M.: Design patterns: Abstraction and reuse of object-oriented design. In: Nierstrasz, O. (ed.) ECOOP 1993. LNCS, vol. 707, pp. 406–431. Springer, Heidelberg (1993)
Mattson, T.G., Sanders, B.A., Massingill, B.L.: Patterns for Parallel Programming. SPS. Addison-Wesley, Reading (2005)
Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. In: OSDI 2004, Sixth Symp. on Operating System Design and Implementation (2004)
Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008)
Lämmel, R.: Google’s MapReduce programming model - Revisited. Science of Computer Programming 70(1), 1–30 (2008)
Poldner, M., Kuchen, H.: Scalable farms. In: Proceedings of ParCo 2005. NIC Series, vol. 33 (2005)
Benoit, A.: ESkel — The Edinburgh Skeleton Library. University of Edinburgh (2007), http://homepages.inf.ed.ac.uk/abenoit1/eSkel/
Michaelson, G., Scaife, N., Bristow, P., King, P.: Nested Algorithmic Skeletons from Higher Order Functions. Parallel Algorithms and Appl. 16, 181–206 (2001)
Blelloch, G.: Programming Parallel Algorithms. Communications of the ACM 39(3), 85–97 (1996)
Danelutto, M., DiCosmo, R., Leroy, X., Pelagatti, S.: Parallel functional programming with skeletons: the OCamlP3L experiment. In: ACM workshop on ML and its applications (1998)
Loogen, R., Ortega-Mallén, Y., Peña-Marí, R.: Parallel Functional Programming in Eden. Journal of Functional Programming 15(3), 431–475 (2005)
Chu, C.T., Kim, S.K., Lin, Y.A., Yu, Y., Bradski, G., Ng, A.Y., Olukotun, K.: Map-reduce for machine learning on multicore. In: Schölkopf, B., Platt, J., Hoffman, T. (eds.) Advances in Neural Information Processing Systems, vol. 19. MIT Press, Cambridge (2007)
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyraki, C.: Evaluating mapreduce for multi-core and multiprocessor systems. In: HPCA 2007: Int. Symp. on High Performance Computer Architecture, pp. 13–24. IEEE Computer Society, Los Alamitos (2007)
Bayley, D., Barszcz, E., et al.: NAS Parallel Benchmarks. Technical Report RNR-94-007, NASA Advanced Supercomputing (NAS) Division (1994)
Apache Hadoop Project. Web page, http://hadoop.apache.org
de Kruijf, M., Sankaralingam, K.: MapReduce for the Cell B.E. Architecture. Technical Report TR1625, Computer Sc.Dept., Madison, WI (2007)
Yeung, J.H., Tsang, C., Tsoi, K., Kwan, B.S., Cheung, C.C., Chan, A.P., Leong, P.H.: Map-reduce as a programming model for custom computing machines. In: Proc. IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM). IEEE Computer Society, Los Alamitos (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Berthold, J., Dieterle, M., Loogen, R. (2009). Implementing Parallel Google Map-Reduce in Eden. In: Sips, H., Epema, D., Lin, HX. (eds) Euro-Par 2009 Parallel Processing. Euro-Par 2009. Lecture Notes in Computer Science, vol 5704. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03869-3_91
Download citation
DOI: https://doi.org/10.1007/978-3-642-03869-3_91
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03868-6
Online ISBN: 978-3-642-03869-3
eBook Packages: Computer ScienceComputer Science (R0)