survey

Kubernetes Scheduling: Taxonomy, Ongoing Issues and Challenges

Author:
Carmen Carrión

School of Computer Science and Engineering, University of Castilla-La, Albacete, Spain

School of Computer Science and Engineering, University of Castilla-La, Albacete, Spain

0000-0002-7481-2226
View Profile

Authors Info & Claims

ACM Computing Surveys Volume 55 Issue 7Article No.: 138pp 1–37https://doi.org/10.1145/3539606

Published:15 December 2022Publication History

ACM Computing Surveys

Abstract

Continuous integration enables the development of microservices-based applications using container virtualization technology. Container orchestration systems such as Kubernetes, which has become the de facto standard, simplify the deployment of container-based applications. However, developing efficient and well-defined orchestration systems is a challenge.

This article focuses specifically on the scheduler, a key orchestrator task that assigns physical resources to containers. Scheduling approaches are designed based on different Quality of Service (QoS) parameters to provide limited response time, efficient energy consumption, better resource utilization, and other things. This article aims to establish insight knowledge into Kubernetes scheduling, find the main gaps, and thus guide future research in the area. Therefore, we conduct a study of empirical research on Kubernetes scheduling techniques and present a new taxonomy for Kubernetes scheduling. The challenges, future direction, and research opportunities are also discussed.

REFERENCES

[1] Adhikari M., Amgoth T., and Srirama S. N.. 2019. A survey on scheduling strategies for workflows in cloud environment and emerging trends. Computing Surveys 52, 4, Article 68 (Aug.2019), 36 pages. Google Scholar
[2] Ahmad I., AlFailakawi M. Gh., AlMutawa A., and Alsalman L.. 2021. Container scheduling techniques: A Survey and assessment. Journal of King Saud University - Computer and Information Sciences 34, 7 (2021), 3934–3947. Google Scholar
[3] Ambrosino G., Fioccola G. B., Canonico R., and Ventre G.. 2020. Container mapping and its impact on performance in containerized cloud environments. In Proceedings of the 2020 IEEE International Conference on Service Oriented Systems Engineering.Google ScholarCross Ref
[4] Arunarani A. R., Manjula D., and Sugumaran Vijayan. 2019. Task scheduling techniques in cloud computing: A literature survey. Future Generation Computer Systems 91 (2019), 407–415. Google ScholarDigital Library
[5] authors The Docker. 2022. Empowering app development for developer. Docker Homepage. Retrieved from https://www.docker.com/.Google Scholar
[6] authors The Docker. 2022. Swarm mode overview. Docker Swarm Homepage. Retrieved from https://docs.docker.com/engine/swarm/.Google Scholar
[7] authors The Kubernetes. 2022. Kubernetes: Production-Grade Container Orchestration. Kubernetes Homepage. Retrieved from http://kubernetes.io/.Google Scholar
[8] Bader J., Thamsen L., Kulagina S., Will J., Meyerhenke H., and Kao O.. 2021. Tarema: Adaptive resource allocation for scalable scientific workflows in heterogeneous clusters. In Proceedings of the 2021 IEEE International Conference on Big Data. IEEE, 65–75.Google ScholarCross Ref
[9] Bao Yixin, Peng Yanghua, and Wu Chuan. 2019. Deep learning-based job placement in distributed machine learning clusters. In Proceedings of the IEEE Conference on Computer Communications. 505–513.Google ScholarDigital Library
[10] Bao Yixin, Peng Yanghua, Wu Chuan, and Li Zongpeng. 2018. Online job scheduling in distributed machine learning clusters. In Proceedings of the IEEE Conference on Computer Communications.Google ScholarDigital Library
[11] Beltre Angel, Saha Pankaj, and Govindaraju Madhusudhan. 2019. KubeSphere: An approach to multi-tenant fair scheduling for kubernetes clusters. In Proceedings of the 2019 IEEE Cloud Summit. IEEE, 14–20.Google ScholarCross Ref
[12] Bentaleb Ouafa, Belloum Adam S. Z., Sebaa Abderrazak, and El-Maouhab Aouaouche. 2022. Containerization technologies: Taxonomies, applications and challenges. The Journal of Supercomputing 78, 1 (2022), 1144–1181.Google ScholarDigital Library
[13] Bestari M., Kistijantoro A., and Sasmita A.. 2020. Dynamic resource scheduler for distributed deep learning training in kubernetes. In Proceedings of the 2020 7th International Conference on Advance Informatics: Concepts, Theory and Applications. 1–6.Google ScholarCross Ref
[14] Biran Yahav, Pasricha Sudeep, Collins George, and Dubow Joel. 2016. Enabling green content distribution network by cloud orchestration. In Proceedings of the 2016 3rd Smart Cloud Networks Systems. 1–8.Google ScholarCross Ref
[15] Bonomi F., Milito R., Zhu J., and Addepalli S.. 2012. Fog computing and its role in the internet of things. In Proceedings of the 1st Edition of the MCC Workshop on Mobile Cloud Computing. ACM, New York, NY. Google Scholar
[16] Borgia E.. 2014. The Internet of Things vision: Key features, applications and open issues. Computer Communications 54 (2014), 1–31. Google ScholarDigital Library
[17] Bulej L., Bureš T., Hnětynka P., and Khalyeyev D.. 2021. Self-adaptive K8S cloud controller for time-sensitive applications. In Proceedings of the 2021 47th Euromicro Conference on Software Engineering and Advanced Applications. 166–169.Google ScholarCross Ref
[18] Burns B., Grant B., Oppenheimer D., Brewer E., and Wilkes J.. 2016. Borg, omega, and kubernetes. Communications of the ACM 59, 5 (42016), 50–57.Google ScholarDigital Library
[19] Caminero A. C. and Muñoz-Mansilla R.. 2021. Quality of service provision in fog computing: Network-aware scheduling of containers. Sensors 21, 12 (2021), 3978. Google ScholarCross Ref
[20] Carvalho M. and Macedo D. F.. 2021. QoE-aware container scheduler for co-located cloud environments. In Proceedings of the 2021 IFIP/IEEE International Symposium on Integrated Network Management. 286–294.Google Scholar
[21] Casalicchio E.. 2019. Container Orchestration: A Survey. Springer Int. Publishing, Cham, 221–235. Google Scholar
[22] Casalicchio E. and Iannucci S.. 2020. The state-of-the-art in container technologies: Application, orchestration and security. Concurrency and Computation: Practice and Experience 32, 17 (2020), e5668.Google ScholarCross Ref
[23] Casquero O., Armentia A., Sarachaga I., F. Perez, D. Orive, and M. Marcos. 2019. Distributed scheduling in Kubernetes based on MAS for fog-in-the-loop applications. In Proceedings of the 24th IEEE International Conference on Emerging Technologies and Factory Automation. 1213–1217. Google ScholarDigital Library
[24] Chang C., Yang S., Yeh E., Lin P., and Jeng J.. 2017. A kubernetes-based monitoring platform for dynamic cloud resource provisioning. In Proceedings of the 2017 IEEE Global Communications Conference1–6.Google ScholarDigital Library
[25] Chhajed Saurabh. 2015. Learning ELK Stack. Packt. Google Scholar
[26] authors The Google cloud. 2021. Container Registry. Container Registry Homepage. Retrieved from https://cloud.google.com/container-registry/.Google Scholar
[27] CNCF. 2022. Cloud Native Computing Foundation Charter. Retrieved from https://www.cncf.io/about/charter/.Google Scholar
[28] Coté M.. 2020. Why Large Organizations Trust Kubernetes. Retrieved from https://tanzu.vmware.com/content/blog/why-large-organizations-trust-kubernetes.Google Scholar
[29] Cérin C., Menouer T., Saad W., and Abdallah W.. 2017. A new docker swarm scheduling strategy. In Proceedings of the 2017 IEEE 7th International Symposium on Cloud and Service Computing. 112–117.Google ScholarCross Ref
[30] Deng S., Zhao H., Xiang Z., C. Zhang, R. Jiang, Y. Li, J. Yin, S. Dustdar, and A. Y. Zomaya. 2022. Dependent function embedding for distributed serverless edge computing. IEEE Transactions on Parallel and Distributed Systems 33, 10 (2022), 2346–2357.Google ScholarCross Ref
[31] Pierro M. Di. 2017. What is the blockchain? Computing in Science Engineering 19, 5 (2017), 92–95.Google ScholarDigital Library
[32] Diouf G. M., Elbiaze H., and Jaafar W.. 2020. On byzantine fault tolerance in multi-master kubernetes clusters. Future Generation Computer Systems 109 (2020), 407–419. Google ScholarCross Ref
[33] Djemame K., Parker M., and Datsev D.. 2020. Open-source serverless architectures: An evaluation of apache openwhisk. In Proceedings of the 2020 IEEE/ACM 13th International Conference on Utility and Cloud Computing. 329–335.Google ScholarCross Ref
[34] Dua A., Randive S., Agarwal A., and Kumar N.. 2020. Efficient load balancing to serve heterogeneous requests in clustered systems using kubernetes. In Proceedings of the 2020 IEEE 17th Annual Consumer Communications & Networking Conference. 1–2.Google ScholarDigital Library
[35] Ahmed G. El Haj, Gil-Castiñeira F., and Costa-Montenegro E.. 2021. KubCG: A dynamic Kubernetes scheduler for heterogeneous clusters. Software: Practice and Experience 51, 2 (2021), 213–234.Google ScholarCross Ref
[36] Ermolenko D., Kilicheva C., Muthanna A., and Khakimov A.. 2021. Internet of Things services orchestration framework based on kubernetes and edge computing. In Proceedings of the 2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering. 12–17.Google ScholarCross Ref
[37] Fan D. and He D.. 2020. Knative autoscaler optimize based on double exponential smoothing. In Proceedings of the 2020 IEEE 5th Information Technology and Mechatronics Engineering Conference. 614–617.Google ScholarCross Ref
[38] Faticanti F., Santoro D., Cretti S., and Siracusa D.. 2021. An application of kubernetes cluster federation in fog computing. In Proceedings of the 2021 24th Conference on Innovation in Clouds, Internet and Networks and Workshops. 89–91.Google ScholarCross Ref
[39] Fayos-Jordan R., Felici-Castell S., Segura-Garcia J., Lopez-Ballester J., and Cobos M.. 2020. Performance comparison of container orchestration platforms with low cost devices in the fog, assisting Internet of Things applications. Journal of Network and Computer Applications 169 (2020), 102788. Google ScholarCross Ref
[40] Foundation The Apache Software. 2022. Apache Hadoop YARN. In Apache Hadoop 3.3.1; Apache Hadoop YARN. Retrieved from https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html.Google Scholar
[41] Foundation The Linux. 2022. Open Container Initiative - OCI. Retrieved from https://opencontainers.org/.Google Scholar
[42] Fu Y., Zhang S., Terrero J., Y. Mao, G. Liu, S. Li, and D. Tao. 2019. Progress-based container scheduling for short-lived applications in a kubernetes cluster. In Proceedings of the 2019 IEEE International Conference on Big Data. 278–287.Google ScholarCross Ref
[43] Ghobaei-Arani M., Souri A., and Rahmanian A.. 2020. Resource management approaches in fog computing: A comprehensive review. Journal of Grid Computing 18, 1 (2020), 1–42.Google ScholarCross Ref
[44] Ghodsi A., Zaharia M., Hindman B., Konwinski A., Shenker S., and Stoica I.. 2011. Dominant resource fairness: Fair allocation of multiple resource types. In Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation.Google Scholar
[45] Govind H. and González-Vélez H.. 2021. Benchmarking serverless workloads on kubernetes. In Proceedings of the 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing. 704–712.Google ScholarCross Ref
[46] Han Y., Shen S., Wang X., Wang S., and Leung V.. 2021. Tailored learning-based scheduling for kubernetes-oriented edge-cloud system. In Proceedings of the IEEE Conference on Computer Communications. 1–10.Google ScholarDigital Library
[47] Harichane I., Makhlouf A., and Belalem G.. 2020. A proposal of kubernetes scheduler using machine-learning on CPU/GPU cluster. In Intelligent Algorithms in Software Engineering. Silhavy Radek (Ed.), Springer Int. Publishing, 567–580. Google ScholarCross Ref
[48] Hindman B., Konwinski A., Zaharia M., A. Ghodsi, A. D. Joseph, R. Katz, S. Shenker, and I. Stoica. 2011. Mesos: A platform for fine-grained resource sharing in the data center. In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. 295–308.Google ScholarDigital Library
[49] Hong Cheol-Ho and Varghese Blesson. 2019. Resource management in fog/edge computing: A survey on architectures, infrastructure, and algorithms. ACM Computing Surveys 52, 5 (2019), 1–37. DOI:Google ScholarDigital Library
[50] Huang J., Xiao C., and Wu W.. 2020. RLSK: A job scheduler for federated kubernetes clusters based on reinforcement learning. In Proceedings of the 2020 IEEE International Conference on Cloud Engineering. 116–123.Google ScholarCross Ref
[51] Huaxin S., Gu X., Ping K., and Hongyu H.. 2020. An improved kubernetes scheduling algorithm for deep learning platform. In Proceedings of the 2020 17th International Computer Conference onWavelet Active Media Technology and Information Processing. 113–116. Google ScholarCross Ref
[52] Hussain K., Salleh M. N. Mohd, Cheng S., and Shi Y.. 2019. Metaheuristic research: A comprehensive survey. Artificial Intelligence Review 52, 4 (2019), 2191–2233.Google ScholarDigital Library
[53] K8. 2022. sig-scheduling blob. Github Repository. 53 pages. Retrieved from https://bit.ly/3jbwx5O.Google Scholar
[54] Kang R., Zhu M., He F., Sato T., and Oki E.. 2021. Design of scheduler plugins for reliable function allocation in kubernetes. In Proceedings of the 2021 17th International Conference on the Design of Reliable Communication Networks. 1–3.Google ScholarCross Ref
[55] Katenbrink F., Seitz A., Mittermeier L., Müller H., and Bruegge B.. 2018. Dynamic scheduling for seamless computing. In Proceedings of the 2018 IEEE 8th International Symposium on Cloud and Service Computing. 41–48.Google ScholarCross Ref
[56] Kaur K., Garg S., Kaddoum G., S. H. Ahmed, and M. Atiquzzaman. 2020. KEIDS: Kubernetes-based energy and interference driven scheduler for industrial IoT in edge-cloud ecosystem. IEEE Internet of Things Journal 7, 5 (May2020), 4228–4237. Google ScholarCross Ref
[57] Kayal Paridhika. 2020. Kubernetes in fog computing: Feasibility demonstration, limitations and improvement scope: Invited paper. In Proceedings of the 2020 IEEE 6th World Forum on Internet of Things. 1–6.Google ScholarCross Ref
[58] Kim E., Lee K., and Yoo C.. 2021. On the resource management of kubernetes. In Proceedings of the 2021 International Conference on Information Networking. 154–158.Google ScholarCross Ref
[59] Kitchenham. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report EBSE-2007-01. 53 pages. Retrieved from https://bit.ly/3t40kAY.Google Scholar
[60] Kumar M., Sharma S. C., Goel A., and Singh S. P.. 2019. A comprehensive survey for scheduling techniques in cloud computing. Journal of Network and Computer Applications 143 (2019), 1–33. Google ScholarDigital Library
[61] Li W., Cao S., Hu K., Cao J., and Buyya R.. 2021. Blockchain-enhanced fair task scheduling for cloud-fog-edge coordination environments: Model and algorithm. Security and Communication Networks 2021 (2021), 1–18.Google Scholar
[62] Li X., Jiang Y., Ding Y., Wei D., Ma X., and Li W.. 2020. Application research of docker based on mesos application container cluster. In Proceedings of the 2020 International Conference on Computer Vision, Image and Deep Learning. 476–479.Google ScholarCross Ref
[63] Li Zijun, Guo L., Cheng J., Chen Q., He B., and Guo M.. 2021. The serverless computing survey: A technical primer for design architecture. ACM Computing Survey (Dec. 2021). Google Scholar
[64] Lin Chan-Yi, Yeh Ting-An, and Chou Jerry. 2019. DRAGON: A dynamic scheduling and scaling controller for managing distributed deep learning jobs in kubernetes cluster. In 9th International Conference on Cloud Computing and Services Science. 569–577.Google ScholarDigital Library
[65] Lin Miao, Xi Jianqing, Bai Weihua, and Wu Jiayin. 2019. Ant colony algorithm for multi-objective optimization of container-based microservice scheduling in cloud. IEEE Access 7 (62019), 83088–83100.Google ScholarCross Ref
[66] Ling W., Ma L., Tian C., and Hu Z.. 2019. Pigeon: A dynamic and efficient serverless and faas framework for private cloud. In Proceedings of the 2019 International Conference on Computational Science and Computational Intelligence. 1416–1421.Google ScholarCross Ref
[67] Luong D., Outtagarts A., and Ghamri-Doudane Y.. 2019. Multi-level resource scheduling for network slicing toward 5G. In Proceedings of the 2019 10th International Conference on Networks of the Future. 25–31.Google ScholarCross Ref
[68] Mahmud R., Ramamohanarao K., and Buyya R.. 2020. Application management in fog computing environments: A taxonomy, review and future directions. Computing Surveys 53, 4, Article 88 (July2020), 43 pages. Google Scholar
[69] Manvi S. and Shyam G. Krishna. 2014. Resource management for infrastructure as a service (IaaS) in cloud computing: A survey. Journal of Network and Computer Applications 41 (2014), 424–440. Google ScholarCross Ref
[70] Mao Y., Fu Y., Zheng W., Cheng L., Liu Q., and Tao D.. 2021. Speculative container scheduling for deep learning applications in a kubernetes cluster. IEEE Systems Journal (2021), 1–12.Google Scholar
[71] Matrouk Khaled and Alatoun Kholoud. 2021. Scheduling algorithms in fog computing: A survey. International Journal of Networked and Distributed Computing 9, 1 (2021), 59–74.Google ScholarCross Ref
[72] Medel Víctor, Rana Omer, Bañares José Ãngel, and Arronategui Unai. 2016. Adaptive application scheduling under interference in kubernetes. In Proceedings of the 2016 IEEE/ACM 9th International Conference on Utility and Cloud Computing. 426–427.Google ScholarDigital Library
[73] Medel V., Tolón C., Arronategui U., Tolosana-Calasanz R., Bañares J. A., and Rana O. F.. 2017. Client-side scheduling based on application characterization on kubernetes. In Proceedings of the International Conference on the Economics of Grids, Clouds, Systems, and Services. 162–176. Google ScholarCross Ref
[74] Menouer Tarek. 2021. KCSS: Kubernetes container scheduling strategy. The Journal of Supercomputing 77, 5 (2021), 4267–4293.Google ScholarCross Ref
[75] Misale C., Drocco M., Milroy D. J., Gutierrez C., S. Herbein, D. H. Ahn, and Y. Park. 2021. It’s a scheduling affair: PolarisACS in the cloud with the kubeflux scheduler. In Proceedings of the 3rd International WS on Containers and New Orchestration Paradigms for Isolated Env. in HPC. 10–16.Google Scholar
[76] Mohanty S., Premsankar G., and Francesco M. di. 2018. An evaluation of open source serverless computing frameworks. In Proceedings of the 2018 IEEE International Conference on Cloud Computing Technology and Science. 115–120.Google ScholarCross Ref
[77] Mon Myat Myat and Khine May Aye. 2019. Scheduling and load balancing in cloud-fog computing using swarm optimization techniques: A survey. In Proceedings of the 17th International Conference on Computer Applications. 8–14.Google Scholar
[78] Mondal S., Pan R., Kabir H. M., Tian T., and Dai H.. 2022. Kubernetes in IT administration and serverless computing: An empirical study and research challenges. The Journal of Supercomputing 78, 2 (2022), 2937–2987.Google ScholarDigital Library
[79] Monteiro L., Almeida W. H., Hazin R., A. C. de Lima, S. K. G. e Silva, and F. S. Ferraz. 2018. A survey on microservice security-trends in architecture, privacy and standardization on cloud computing environments. International Journal on Advances in Security 11, 3–4 (2018), 201–213.Google Scholar
[80] Morabito R.. 2016. A performance evaluation of container technologies on Internet of Things devices. In Proceedings of the 2016 IEEE Conference on Computer Communications Workshops. 999–1000.Google ScholarCross Ref
[81] Moravcik M. and Kontsek M.. 2020. Overview of docker container orchestration tools. In Proceedings of the 2020 18th International Conference on Emerging eLearning Technologies and Applications. 475–480.Google ScholarCross Ref
[82] Musaddiq A., Zikria Y., Hahm O., Yu H., Bashir A., and Kim S.. 2018. A survey on resource management in IoT operating systems. IEEE Access 6 (2018), 8459–8482.Google ScholarCross Ref
[83] Nastic S., Pusztai T., Morichetta A., Pujol V., Dustdar S., Vii D., and Xiong Y.. 2021. Polaris scheduler: Edge sensitive and SLO aware workload scheduling in cloud-edge-IoT clusters. In Proceedings of the IEEE 14th International Conference on Cloud Computing. 206–216.Google ScholarCross Ref
[84] Nguyen K., Drew S., Huang C., and Zhou J.. 2020. Collaborative container-based parked vehicle edge computing framework for online task offloading. In Proceedings of the 2020 IEEE 9th International Conference on Cloud Networking. 1–6.Google ScholarCross Ref
[85] Nguyen N. D., Phan L., Park D., Kim S., and Kim T.. 2020. ElasticFog: Elastic resource provisioning in container-based fog computing. IEEE Access 8 (2020), 183879–183890.Google ScholarCross Ref
[86] Nguyen T., Yeom Y., Kim T., Park D., and Kim S.. 2020. Horizontal pod autoscaling in kubernetes for elastic container orchestration. Sensors 20, 16 (2020), 4621. Google ScholarCross Ref
[87] Núñez-Gómez C., Caminero B., and Carrión C.. 2021. HIDRA: A distributed blockchain-based architecture for fog/edge computing environments. IEEE Access 9 (2021), 75231–75251.Google Scholar
[88] Ogbuachi M. C., Gore C., Reale A., P. Suskovics, and B. Kovacs. 2019. Context-aware K8S scheduler for real time distributed 5G edge computing applications. In Proceedings of the 2019 International Conference on Software, Telecommunications and Computer Networks. 1–6. Google ScholarCross Ref
[89] Pahl C., Brogi A., Soldani J., and Jamshidi P.. 2019. Cloud container technologies: A state-of-the-art review. IEEE Transactions on Cloud Computing 7, 3 (2019), 677–692.Google Scholar
[90] Peng Y., Bao Y., Chen Y., Wu C., Meng C., and Lin W.. 2021. DL2: A deep learning-driven scheduler for deep learning clusters. IEEE Transactions on Parallel and Distributed Systems 32, 8 (2021), 1947–1960.Google ScholarCross Ref
[91] Perera C., Qin Y., Estrella J., Reiff-Marganiec S., and Vasilakos A.. 2017. Fog computing for sustainable smart cities: A survey. Computing Surveys 50, 3, Article 32 (June2017), 43 pages. Google ScholarDigital Library
[92] Prado R. P., Galán S. G., Expósito J., Marchewka A., and Ruiz-Reyes N.. 2020. Smart containers schedulers for microservices provision in cloud-fog-IoT networks. challenges and opportunities. Sensors 20, 6 (2020), 1714. Google Scholar
[93] authors 2022 K3s project. 2022. Lightweight Kubernetes: The Certified Kubernetes Distribution Built for IoT and Edge Computing. Retrieved from https://k3s.io/.Google Scholar
[94] authors KubeEdge project. 2021. A Kubernetes Native Edge Computing Framework. Retrieved from https://kubeedge.io/en/.Google Scholar
[95] authors © Kubeless 2022 project. 2022. The Kubernetes Native Serverless Framework. Retrieved from https://kubeless.io/.Google Scholar
[96] Pusztai T., Rossi F., and Dustdar S.. 2021. Pogonip: Scheduling asynchronous applications on the edge. In Proceedings of the 2021 IEEE 14th International Conference on Cloud Computing. 660–670.Google ScholarCross Ref
[97] Qi S., Kulkarni S., and Ramakrishnan K.. 2021. Assessing container network interface plugins: Functionality, performance, and scalability. IEEE Transactions on Network and Service Management 18, 1 (2021), 656–671.Google ScholarDigital Library
[98] Qi S., Kulkarni S. G., and Ramakrishnan K. K.. 2020. Understanding container network interface plugins: Design considerations and performance. In Proceedings of the 2020 IEEE International Symposium on Local and Metropolitan Area Networks. 1–6.Google ScholarCross Ref
[99] Rahali M., Phan C., and Rubino G.. 2021. KRS: Kubernetes resource scheduler for resilient NFV networks. In Proceedings of the IEEE Global Communications Conference.IEEE, Madrid, 1–6.Google ScholarCross Ref
[100] Rashid Aaqib and Chaturvedi Amit. 2019. Virtualization and its role in cloud computing environment. International Journal of Computer Sciences and Engineering 7, 4 (2019), 1131–1136.Google ScholarCross Ref
[101] Rattihalli G., Govindaraju M., Lu H., and Tiwari D.. 2019. Exploring potential for non-disruptive vertical auto scaling and resource estimation in kubernetes. In Proceedings of the 2019 IEEE 12th International Conference on Cloud Computing. 33–40.Google ScholarCross Ref
[102] Rocha I., Göttel C., Felber P., M. Pasin, R. Rouvoy, and V. Schiavoni. 2019. Heats: Heterogeneity-and energy-aware task-based scheduling. In Proceedings of the 2019 27th Euromicro International Conference on Parallel, Distributed and Network-Based Processing. 400–405.Google ScholarCross Ref
[103] Rodriguez M. A. and Buyya R.. 2019. Container-based cluster orchestration systems: A taxonomy and future directions. Software: Practice and Experience 49, 5 (2019), 698–719.Google ScholarCross Ref
[104] Sami H., Mourad A., Otrok H., and Bentahar J.. 2020. FScaler: Automatic resource scaling of containers in fog clusters using reinforcement learning. In Proceedings of the 2020 International Wireless Communications and Mobile Computing. 1824–1829.Google ScholarCross Ref
[105] Santoro D., Zozin D., Pizzolli D., Pellegrini F. De, and Cretti S.. 2017. Foggy: A platform for workload orchestration in a fog computing environment. In Proceedings of the 2017 IEEE International Conference on Cloud Computing Technology and Science. 231–234.Google ScholarCross Ref
[106] Santos J., Wauters T., Volckaert B., and Turck F. De. 2019. Resource provisioning in fog computing: From theory to practice. Sensors 19, 10 (2019), 2238. Google ScholarCross Ref
[107] Santos J., Wauters T., Volckaert B., and Turck F. De. 2019. Towards network-aware resource provisioning in kubernetes for fog computing applications. In Proceedings of the 2019 IEEE Conference on Network Softwarization. 351–359.Google ScholarCross Ref
[108] Santos J., Wauters T., Volckaert B., and Turck F. De. 2020. Towards delay-aware container-based service function chaining in fog computing. In Proceedings of the 2020 IEEE/IFIP Network Operations and Management Symposium. 1–9.Google ScholarDigital Library
[109] Scazzariello M., Ariemma L., Battista G., and Patrignani M.. 2020. Megalos: A scalable architecture for the virtualization of network scenarios. In Proceedings of the 2020 IEEE/IFIP Network Operations and Management Symposium.1–7.Google ScholarDigital Library
[110] Schwarzkopf M., Konwinski A., Abd-El-Malek M., and Wilkes J.. 2013. Omega: Flexible, scalable schedulers for large compute clusters. In Proceedings of the 8th ACM European Conference on Computer Systems. 351–364. Google ScholarDigital Library
[111] Sfakianakis Y., Marazakis M., and Bilas A.. 2021. Skynet: Performance-driven resource management for dynamic workloads. In Proceedings of the 2021 IEEE 14th International Conference on Cloud Computing. 527–539.Google ScholarCross Ref
[112] Shahrad M., Balkind J., and Wentzlaff D.. 2019. Architectural implications of function-as-a-service computing. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture. 1063–1075. Google ScholarDigital Library
[113] Sharma P., Chaufournier L., Shenoy P., and Tay Y. C.. 2016. Containers and virtual machines at scale: A comparative study. In Proceedings of the 17th international Middleware Conference. New York, NY, Article 1, 13 pages. Google ScholarDigital Library
[114] Song S., Deng L., Gong J., and Luo H.. 2018. Gaia scheduler: A kubernetes-based scheduler framework. In Proceedings of the 2018 IEEE Intl Conf on Parallel Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications. 252–259.Google ScholarCross Ref
[115] Stanojevic P., Usorac S., and Stanojev N.. 2021. Container manager for multiple container runtimes. In Proceedings of the 2021 44th International Convention on Information, Communication and Electronic Technology. 991–994.Google ScholarCross Ref
[116] Surya R. Y. and Kistijantoro A. Imam. 2019. Dynamic resource allocation for distributed tensorflow training in kubernetes cluster. In Proceedings of the 2019 International Conference on Data and Software Engineering. 1–6.Google ScholarCross Ref
[117] Tamiru M. A., Pierre G., Tordsson J., and Elmroth E.. 2021. mck8s: An orchestration platform for geo-distributed multi-cluster environments. In Proceedings of the 2021 International Conference on Computer Communications and Networks. 1–10.Google ScholarCross Ref
[118] Tang J., Jalalzai M. M., Feng C., Xiong Z., and Zhang Y.. 2022. Latency-aware task scheduling in software-defined edge and cloud computing with erasure-coded storage systems. IEEE Transactions on Cloud Computing (2022), 1–1.Google Scholar
[119] Tantawi Asser N. and Steinder Malgorzata. 2019. Autonomic cloud placement of mixed workload: An adaptive bin packing algorithm. In Proceedings of the 2019 IEEE International Conference on Autonomic Computing. 187–193.Google ScholarCross Ref
[120] Thinakaran P., Gunasekaran J. R., Sharma B., M. T. Kandemir, and C. R. Das. 2019. Kube-Knots: Resource harvesting through dynamic container orchestration in GPU-based datacenters. In Proceedings of the 2019 IEEE International Conference on Cluster Comp.1–13.Google ScholarCross Ref
[121] Toka L.. 2021. Ultra-reliable and low-latency computing in the edge with kubernetes. Journal of Grid Computing 19, 3 (2021), 1–23.Google ScholarDigital Library
[122] Townend P., Clement S., Burdett D., R. Yang, J. Shaw, B. Slater, and J. Xu. 2019. Invited paper: Improving data center efficiency through holistic scheduling in kubernetes. In Proceedings of the 2019 IEEE International Conference on Service-Oriented System Engineering. 156–15610. Google ScholarCross Ref
[123] Tzenetopoulos A., Masouros D., Xydis S., and Soudris D.. 2020. Interference-aware orchestration in kubernetes. In Proceedings of the International Conference on High Performance Computing. 12321.Google Scholar
[124] Vaucher S., Pires R., Felber P., Pasin M., Schiavoni V., and Fetzer C.. 2018. SGX-aware container orchestration for heterogeneous clusters. In Proceedings of the 2018 IEEE 38th International Conference on Distributed Computing Systems. 730–741.Google ScholarCross Ref
[125] Vayghan L., Saied M., Toeroe M., and Khendek F.. 2019. Microservice based architecture: Towards high-availability for stateful applications with kubernetes. In Proceedings of the 2019 IEEE 19th International Conference on Software Quality, Reliability and Security. 176–185.Google Scholar
[126] Verma A., Pedrosa L., Korupolu M., Oppenheimer D., Tune E., and Wilkes J.. 2015. Large-scale cluster management at Google with borg. In Proceedings of the 10th European Conference on Computer Systems. Article 18, 17 pages. Google ScholarDigital Library
[127] Wang S., Gonzalez O. J., Zhou X., T. Williams, B. D. Friedman, M. Havemann, and T. Woo. 2020. An efficient and non-intrusive GPU scheduling framework for deep learning training systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–13.Google ScholarDigital Library
[128] Warke A., Mohamed M., Engel R., Ludwig H., Sawdon W., and Liu L.. 2018. Storage service orchestration with container elasticity. In Proceedings of the 2018 IEEE 4th International Conference on Collaboration and Internet Computing. 283–292.Google ScholarCross Ref
[129] Wojciechowski Ł., Opasiak K., Latusek J., M. Wereski, V. Morales, T. Kim, and M. Hong. 2021. NetMARKS: Network metrics-aware kubernetes scheduler powered by service mesh. In Proceedings of the IEEE Conference on Computer Communications. 1–9.Google ScholarDigital Library
[130] XIE X. and Govardhan S. S.. 2020. A service mesh-based load balancing and task scheduling system for deep learning applications. In Proceedings of the 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing. 843–849.Google ScholarCross Ref
[131] Xiong Ying, Sun Yulin, Xing Li, and Huang Ying. 2018. Extend cloud to edge with kubeedge. In Proceedings of the 2018 IEEE/ACM Symposium on Edge Computing. 373–377.Google ScholarCross Ref
[132] Yang S., Ren Y., Zhang J., Guan J., and Li B.. 2021. KubeHICE: Performance-aware container orchestration on heterogeneous-ISA architectures in cloud-edge platforms. In Proceedings of the 2021 IEEE International Conference on Parallel, Distributed Processing with Apps, Big Data, Cloud Computing, Sustainable Computing, Communications, Social Computing, Net.81–91.Google ScholarCross Ref
[133] Yang Y. and Chen L.. 2019. Design of kubernetes scheduling strategy based on LSTM and grey model. In Proceedings of the 2019 IEEE 14th International Conference on Intelligent Systems and Knowledge Engineering. 701–707.Google ScholarCross Ref
[134] Yilmaz Onur. 2021. Extending the kubernetes API. In Extending Kubernetes. Springer, 99–141.Google ScholarCross Ref
[135] Yousefpour A., Fung C., Nguyen T., K. Kadiyala, F. Jalali, A. Niakanlahiji, J. Kong, and J. P. Jue. 2019. All one needs to know about fog computing and related edge computing paradigms: A complete survey. Journal Systems Architecture 98 (2019), 289—330. Google ScholarDigital Library
[136] Yu Dongjin, Jin Yike, Zhang Yuqun, and Zheng Xi. 2018. A survey on security issues in services communication of microservices-enabled fog applications. Concurrency and Computation: Practice and Experience 0, 0 (2018), e4436.Google Scholar
[137] Zhang G., Lu R., and Wu W.. 2019. Multi-resource fair allocation for cloud federation. In Proceedings of the 2019 IEEE 21st International Conference on HP Computing and Communications; 17th International Conference on Smart City; 5th International Conference on Data Science and Systems. 2189–2194.Google ScholarCross Ref
[138] Zhang X., Li L., Wang Y., Chen E., and Shou L.. 2021. Zeus: Improving resource efficiency via workload colocation for massive kubernetes clusters. IEEE Access 9 (2021), 105192–105204.Google Scholar
[139] Zhao H., Deng S., Liu Z., Yin J., and Dustdar S.. 2022. Distributed redundancy scheduling for microservice-based applications at the edge. IEEE Transactions on Services Computing 15, 3 (2022), 1732–1745.Google Scholar
[140] Zheng C., Kremer-Herman N., Shaffer T., and Thain D.. 2020. Autoscaling high-throughput workloads on container orchestrators. In Proceedings of the 2020 IEEE International Conference on Cluster Computing. 142–152.Google ScholarCross Ref

Index Terms

Kubernetes Scheduling: Taxonomy, Ongoing Issues and Challenges

Recommendations

Cost-efficient scheduling algorithms based on beetle antennae search for containerized applications in Kubernetes clouds
Abstract
With the development of cloud-native technologies, Kubernetes becomes the standard of fact for container scheduling. Kubernetes provides service discovery and scheduling of containers, load balancing, service self-healing, elastic scaling, storage ...
Read More
A Systematic Literature Review on Maintenance of Software Containers
Nowadays, cloud computing is gaining tremendous attention to deliver information via the internet. Virtualization plays a major role in cloud computing as it deploys multiple virtual machines on the same physical machine and thus results in improving ...
Read More
Custom Scheduling in Kubernetes: A Survey on Common Problems and Solution Approaches
Since its release in 2014, Kubernetes has become a popular choice for orchestrating containerized workloads at scale. To determine the most appropriate node to host a given workload, Kubernetes makes use of a scheduler that takes into account a set of ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Computing Surveys Volume 55, Issue 7
July 2023
813 pages
ISSN:0360-0300
EISSN:1557-7341
DOI:10.1145/3567472
Editor:
Albert Zomaya
University of Sydney, Australia
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 15 December 2022
- Online AM: 2 June 2022
- Accepted: 23 May 2022
- Revised: 17 May 2022
- Received: 20 October 2021
Published in csur Volume 55, Issue 7

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Kubernetes
orchestration
scheduling
containers
survey
Qualifiers
- survey
- Refereed
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 16
  Total Citations
  View Citations
- 6,148
  Total Downloads
- Downloads (Last 12 months)3,103
- Downloads (Last 6 weeks)427
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

HTML Format

View this article in HTML Format .

View HTML Format

Kubernetes Scheduling: Taxonomy, Ongoing Issues and Challenges

ACM Computing Surveys

Abstract

REFERENCES

Cited By

Index Terms

Recommendations

Cost-efficient scheduling algorithms based on beetle antennae search for containerized applications in Kubernetes clouds

A Systematic Literature Review on Maintenance of Software Containers

Custom Scheduling in Kubernetes: A Survey on Common Problems and Solution Approaches