Chunlei Xu
ABSTRACT
In recent years, driven by hardware technology, the computing power and programmability of GPUs have been rapidly developed. With the characteristics of highly parallel computing, GPUs are no longer limited to daily graphics processing tasks. It begins to involve a wider range of high-performance generalpurpose computing field. One of the hotspots in the field of highperformance parallel computing is MapReduce, a massive data processing framework. Through inexpensive ordinary computer clusters, we can obtain large-scale data computing capabilities that were previously only owned by expensive large servers. However, most existing MapReduce systems run on CPU clusters, and the computing performance of a single node is limited. Therefore, this paper proposes a parallel computing framework based on GPU cluster and MapReduce, and validates the effectiveness of the framework through experiments. Experiments have proven that our framework can complete the work, and it has a significant speedup for large-scale applications.
- Mittal S (2016). A survey of techniques for approximate computing[J]. ACM Computing Surveys (CSUR), 48(4), 62.Google Scholar
Digital Library
- CUDA NVIDIA. CUDA Programming Guide.Google Scholar
- Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung (2003). The Google File System, SOSP 03, New York, NY, USA, pp. 29--43.Google ScholarDigital Library
- Jeffrey Dean and Sanjay Ghemawat (2008). MapReduce: Simplyfied Data Processing on Large Clusters. Communications of the ACM, vol. 51, pp. 107--113.Google ScholarDigital Library
- Apache Hadoop, http://hadoop.apache.org/.Google Scholar
- L Shvachko, Hairong Kuang, S Radia, et al. (2010). The Hadoop Distributed File System. IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1--10.Google Scholar
- HBase, http://hbase.apache.org/.Google Scholar
- Ranger C, Raghuraman R, Penmetsa A, et al. (2007). Evaluating mapreduce for multi-core and multiprocessor systems[C]. High Performance Computer Architure HPCA. IEEE 13th International Symposium on IEEE, 13--24.Google ScholarDigital Library
- Yang C T, Huang C L and Lin C F (2011). Hybrid CUDA, OpenMP, and MPI parallel programming on multicore GPU clusters[J]. Computer Physics Communications, 182(1), 266--269.Google ScholarCross Ref
- He B, Fang W, Luo Q, et al. (2008). Mars: a MapReduce framework on graphics processors[C]. Proceedings of the 17th international conference on Parallel architectures and compilation techniques ACM, 260--269.Google ScholarDigital Library
- Hong C, Chen D, Chen W, et al. (2010). MapCG:writing parallel program protable between CPU and GPU[C]. Proceedings of the 19th international conference on Parallel architectures and compilation techniques. ACM, 217--226.Google ScholarDigital Library
- Xin Miao and Li Hao (2012). An Implementation of GPU Accelerated MapReduce: Using Hadoop with OpenCL for Data- and Compute-Intensive Jobs. In IJCSS, pp. 6--11.Google Scholar
- Styart J A and Qwens J D (2011). Multi-GPU MapReduce on GPU clusters[C]. Parallel & Distributed Processing Symposium (IPDPS), IEEE International. IEEE, 1068--1079.Google ScholarDigital Library
- Heng Gao, Jie Tang and Gangshan Wu (2013). A MapReduce Computing Framework Based on GPU Cluster. IEEE Conference, High Performance Computing and Communications & Embedded and Ubiquitous Computing, pages 1902--1907.Google Scholar
- Mengjun Xie, Kyoung-Don Kang and Basaran C (2013). Moim: A Multi-GPU MapReduce Framework. IEEE 16th International Conference on Computational Science and Engineering (CSE), pp.1279, 1286, 3-5 Dec.Google Scholar
- Yiru Guo, Weiguo Liu, Gong B, Voss G and Muller-Wittig W (2013). GCMR: A GPU Cluster-Based MapReduce Framework for Large- Scale Data Processing. High Performance Computing and Communications & IEEE International Conference on Embedded and Ubiquitous Computing (HPCC_EUC), IEEE 10th International Conference on, pp. 580, 586, 13-15 Nov.Google Scholar
Index Terms
- Parallel Computing Framework Based on MapReduce and GPU Clusters
Recommendations
Using distributed memory parallel computers and GPU clusters for multidimensional Monte Carlo integration
The aim of this paper is to show that the multidimensional Monte Carlo integration can be efficiently implemented on various distributed memory parallel computers and clusters of multicore nodes using recently developed parallel versions of linear ...
Accelerating MapReduce framework on multi-GPU systems
Graphics processors evolve rapidly and promise to support power-efficient, cost, differentiated price-performance, and scalable high performance computing. MapReduce is a well-known distributed programming model to ease the development of applications ...
An Implementation of GPU Accelerated MapReduce: Using Hadoop with OpenCL for Data- and Compute-Intensive Jobs
IJCSS '12: Proceedings of the 2012 International Joint Conference on Service SciencesMapReduce is an efficient distributed computing model for large-scale data processing. However, single-node performance is gradually to be the bottleneck in compute-intensive jobs. This paper presents an approach of MapReduce improvement with GPU ...
Comments