Abstract
Hadoop MapReduce has been proved to be an efficient model for distributed data processing. This model is widely used by different service providers, which create a challenge of maintaining same efficiency and performance level in different systems. One of the most critical problems for this model is how to overcome heterogeneity and scalability in different systems. The decreases of performance in heterogeneous environment occur due to inefficient scheduling of Map and Reduce tasks. Another important problem is how to minimize master node overhead and network traffic created by scheduling algorithm. In this paper, we introduce a lightweight adaptive scheduler in which we provide the classifier with information about jobs requirement and node capabilities. The scheduler classifies jobs into executable and nonexecutable according to the nodes capabilities. Then the scheduler assigns the tasks to appropriate nodes in the cluster to get highest performance.
Keywords
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
J. Dean and S. Ghemawat, “Mapreduce: Simplified data processing on large clusters”, Communications of the ACM, VOL. 51, NO. I, pp. 107–113, 2008.
B. Thirumala Rao, Dr. L S S Reddy “Survey on Improved Scheduling in Hadoop MapReduce in Cloud Environments”, in International Journal of Computer Applications (0975-8887) Volume 34. No. 9, November 2011.
B. Thirumala Rao, V. Krishna Reddy. “Performance Issues of Heterogeneous Hadoop Clusters in Cloud Computing”, Global Journal of Computer Science and Technology, Volume XI, Issue VIII, May 2011.
Dr. J. Aghav and Shyam Deshmukh (2013),“Job Classification for MapReduce Scheduler in Heterogeneous Environment”, IEEE Cloud & Ubiquitous Computing & Emerging Technologies (CUBE), 15–16 Nov. 2013, Page: 26.
Dhok J, Varma V (2010), “Using pattern classification for task assignment in MapReduce”, Proceedings of the 8th IEEE International Conference on Grid and Cooperative Computing, Volume 34. No. 9, November 2011.
M. Zaharia, A. Konwinski, A.D. Joseph, R. Katz, and I. Stoica. “Improving mapreduce performance in heterogeneous environments”,. In Proc. Of USENIX OSDI, 2008.
Y. Yao, J. Tai, B. Sheng, and N. Mi, “Scheduling heterogeneous mapreduce jobs for efficiency improvement in enterprise clusters”, Integrated Network Management (1 M 2(13), 2013 IFlPIIEEE International Symposium on, pp. 872–875, 2013.
K. Kc and K. Anyanwu, “Scheduling Hadoop Jobs to Meet Deadlines”, in Proc. CloudCom, 2010, pp. 388–392.
Rasooli and D. G. Down, “A hybrid scheduling approach for scalable heterogeneous hadoop systems”, IEEE Computer Society, 2012, pp. 1284–1291.
J. S. Manjaly and V. S. Chooralil, ‘‘Tasktracker aware scheduling for hadoop mapreduce”, 2013 Third International Conference on Advances in Computing and Communications, pp. 278–281, Aug. 2013.
M. Hammoud and M. F. Sakr, “Locality-aware reduce task scheduling for Mapreduce”, in Proceedings of the 2011 IEEE Third International Conference on Cloud Computing Technology and Science, ser. CLOUDCOM ‘11. Washington, DC, USA: IEEE Computer Society, 2011, pp. 570–576.
S. Humbetov, “Data-intensive computing with map-reduce and hadoop”, IEEE International Conference on Application of Information and Communication Technologies, 17–19 Oct. 2012, pp. 1–5.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Science+Business Media Singapore
About this paper
Cite this paper
Mohammad Ghoneem, Lalit Kulkarni (2017). An Adaptive MapReduce Scheduler for Scalable Heterogeneous Systems. In: Satapathy, S., Bhateja, V., Joshi, A. (eds) Proceedings of the International Conference on Data Engineering and Communication Technology. Advances in Intelligent Systems and Computing, vol 469. Springer, Singapore. https://doi.org/10.1007/978-981-10-1678-3_57
Download citation
DOI: https://doi.org/10.1007/978-981-10-1678-3_57
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-1677-6
Online ISBN: 978-981-10-1678-3
eBook Packages: EngineeringEngineering (R0)