Kavita Agarwal
Donald E. Porter
ABSTRACT
Containers, or OS-based virtualization, have seen a recent resurgence in deployment. The term "container" is nearly synonymous with "lightweight virtualization", despite a remarkable dearth of careful measurements supporting this notion. This paper contributes comparative measurements and analysis of both containers and hardware virtual machines where the functionality of both technologies intersects. This paper focuses on two important issues for cloud computing: density (guests per physical host) and start-up latency (for responding to load spikes). We conclude that the overall density is highly dependent on the most demanded resource. In many dimensions there are no significant differences, and in other dimensions VMs have significantly higher overheads. A particular contribution is the first detailed analysis of the biggest difference---memory footprint---and opportunities to significantly reduce this overhead.
- KVM and Docker LXC Benchmarking with OpenStack. http://bodenr.blogspot.com/2014/05/kvm-and-docker-lxc-benchmarking-with.html.Google Scholar
- Linux Containers. https://linuxcontainers.org/.Google Scholar
- O. Agesen, J. Mattson, R. Rugina, and J. Sheldon. Software Techniques for Avoiding Hardware Virtualization Exits. In Proceedings of the 2012 USENIX Conference on Annual Technical Conference, USENIX ATC'12, pages 35--35, Berkeley, CA, USA, 2012. USENIX Association. Google ScholarDigital Library
- N. Amit, M. Ben-Yehuda, D. Tsafrir, and A. Schuster. vIOMMU: Efficient IOMMU Emulation. In Proceedings of the 2011 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC'11, pages 6--6, Berkeley, CA, USA, 2011. USENIX Association. Google ScholarDigital Library
- Aufs. http://aufs.sourceforge.net/.Google Scholar
- A. Belay, G. Prekas, A. Klimovic, S. Grossman, C. Kozyrakis, and E. Bugnion. IX: A Protected Dataplane Operating System for High Throughput and Low Latency. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 49--65, Broomfield, CO, Oct. 2014. USENIX Association. Google ScholarDigital Library
- S. Bhattiprolu, E. W. Biederman, S. Hallyn, and D. Lezcano. Virtual servers and checkpoint/restart in mainstream Linux. ACM SIGOPS Operating Systems Review, 42:104--113, July 2008. Google ScholarDigital Library
- N. Bila, E. J. Wright, E. D. Lara, K. Joshi, H. A. Lagar-Cavilla, E. Park, A. Goel, M. Hiltunen, and M. Satyanarayanan. Energy-Oriented Partial Desktop Virtual Machine Migration. ACM Trans. Comput. Syst., 33(1):2:1--2:51, Mar. 2015. Google ScholarDigital Library
- E. Bugnion, S. Devine, K. Govil, and M. Rosenblum. Disco: Running commodity operating systems on scalable multiprocessors. ACM Transactions on Computer Systems (TOCS), 15(4):412--447, 1997. Google ScholarDigital Library
- Canonical. LXD crushes KVM in density and speed. https://insights.ubuntu.com/2015/05/18/lxd-crushes-kvm-in-density-and-speed/, 2015.Google Scholar
- W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An Updated Performance Comparison of Virtual Machines and Linux Containers. Technical Report RC25482(AUS1407001), IBM Research Division, 11501 Burnet Road, Austin, TX, 2014.Google Scholar
- Filebench. http://sourceforge.net/projects/filebench/.Google Scholar
- B. Ford and R. Cox. Vx32: Lightweight user-level sandboxing on the x86. In Proceedings of the USENIX Annual Technical Conference, pages 293--306, 2008. Google ScholarDigital Library
- A. Gordon, N. Amit, N. Har'El, M. Ben-Yehuda, A. Landau, A. Schuster, and D. Tsafrir. ELI: Bare-metal Performance for I/O Virtualization. In Proceedings of the Seventeenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XVII, pages 411--422, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- T. Gröninger. On Statistical Properties of Duplicate Memory Pages. Diploma thesis, System Architecture Group, Karlsruhe Institute of Technology (KIT), Germany, Oct.31 2013. http://os.itec.kit.edu/.Google Scholar
- D. Gupta, S. Lee, M. Vrable, S. Savage, A. C. Snoeren, G. Varghese, G. M. Voelker, and A. Vahdat. Difference engine: Harnessing memory redundancy in virtual machines. Communications of the ACM, 53(10):85--93, 2010. Google ScholarDigital Library
- R. Kapoor, G. Porter, M. Tewari, G. M. Voelker, and A. Vahdat. Chronos: Predictable Low Latency for Data Center Applications. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC '12, pages 9:1--9:14, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- Kernel-based virtual machine. http://www.linux-kvm.org/.Google Scholar
- H. A. Lagar-Cavilla, J. A. Whitney, A. M. Scannell, P. Patchin, S. M. Rumble, E. De Lara, M. Brudno, and M. Satyanarayanan. SnowFlock: Rapid Virtual Machine Cloning for Cloud Computing. In Proceedings of the 4th ACM European conference on Computer systems, pages 1--12. ACM, 2009. Google ScholarDigital Library
- J. Liu, W. Huang, B. Abali, and D. K. Panda. High Performance VMM-bypass I/O in Virtual Machines. In Proceedings of the Conference on USENIX '06 Annual Technical Conference, ATEC '06, pages 3--3, Berkeley, CA, USA, 2006. USENIX Association. Google ScholarDigital Library
- A. Madhavapeddy, R. Mortier, C. Rotsos, D. Scott, B. Singh, T. Gazagnaire, S. Smith, S. Hand, and J. Crowcroft. Unikernels: Library operating systems for the cloud. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2013. Google ScholarDigital Library
- J. N. Matthews, W. Hu, M. Hapuarachchi, T. Deshane, D. Dimatos, G. Hamilton, M. McCabe, and J. Owens. Quantifying the Performance Isolation Properties of Virtualization Systems. In Proceedings of the 2007 Workshop on Experimental Computer Science, ExpCS '07, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- J. Mauro and R. McDougall. Solaris Internals (2Nd Edition). Prentice Hall PTR, Upper Saddle River, NJ, USA, 2006. Google ScholarDigital Library
- M. K. McKusick and G. V. Neville-Neil. The Design and Implementation of the FreeBSD Operating System. Pearson Education, 2004. Google ScholarDigital Library
- D. Meisner, J. Wu, and T. Wenisch. BigHouse: A simulation infrastructure for data center systems. In Performance Analysis of Systems and Software (ISPASS), 2012 IEEE International Symposium on, pages 35--45, April 2012. Google ScholarDigital Library
- C. Metz. Google Embraces Docker, the Next Big Thing in Cloud Computing. WIRED, June 2014. http://www.wired.com/2014/06/eric-brewer-google-docker/.Google Scholar
- K. Miller. Efficient Main Memory Deduplication Through Cross Layer Integration. PhD thesis, Karlsruhe, Karlsruher Institut für Technologie (KIT), Diss., 2014, 2014.Google Scholar
- K. Miller, F. Franz, T. Groeninger, M. Rittinghaus, M. Hillenbrand, and F. Bellosa. KSM++: Using I/O-based hints to make memory-deduplication scanners more efficient. In Proceedings of the ASPLOS Workshop on Runtime Environments, Systems, Layering and Virtualized Environments (RESoLVE'12), 2012.Google Scholar
- K. Miller, F. Franz, M. Rittinghaus, M. Hillenbrand, and F. Bellosa. XLH: More Effective Memory Deduplication Scanners Through Cross-layer Hints. In USENIX Annual Technical Conference, pages 279--290, 2013. Google ScholarDigital Library
- D. G. Murray, H. Steven, and M. A. Fetterman. Satori: Enlightened page sharing. In In Proceedings of the USENIX Annual Technical Conference. Citeseer, 2009. Google ScholarDigital Library
- NIST. National Vulnerability Database. http://nvd.nist.gov/, 2008.Google Scholar
- S. Peter, J. Li, I. Zhang, D. R. K. Ports, D. Woos, A. Krishnamurthy, T. Anderson, and T. Roscoe. Arrakis: The Operating System is the Control Plane. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 1--16, Broomfield, CO, Oct. 2014. USENIX Association. Google ScholarDigital Library
- D. E. Porter, S. Boyd-Wickizer, J. Howell, R. Olinsky, and G. Hunt. Rethinking the library OS from the top down. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 291--304, 2011. Google ScholarDigital Library
- D. Price and A. Tucker. Solaris Zones: Operating system support for consolidating commercial workloads. In Proceedings of the Large Installation System Administration Conference (LISA), pages 241--254, 2004. Google ScholarDigital Library
- N. Regola and J.-C. Ducom. Recommendations for Virtualization Technologies in High Performance Computing. In Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on, pages 409--416, Nov 2010. Google ScholarDigital Library
- P. Sharma and P. Kulkarni. Singleton: system-wide page deduplication in virtual environments. In Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing, pages 15--26. ACM, 2012. Google ScholarDigital Library
- S. Soltesz, H. Pötzl, M. E. Fiuczynski, A. Bavier, and L. Peterson. Container-based Operating System Virtualization: A Scalable, High-performance Alternative to Hypervisors. In Proceedings of the 2Nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, EuroSys '07, pages 275--287, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- M. Stokely and C. Lee. The FreeBSD Handbook, 3rd Edition, Vol 1: Users's Guide, 2003. Google ScholarDigital Library
- C.-C. Tsai, K. S. Arora, N. Bandi, B. Jain, W. Jannen, J. John, H. A. Kalodner, V. Kulkarni, D. Oliveira, and D. E. Porter. Cooperation and Security Isolation of Library OSes for Multi-Process Applications. In Proceedings of the ACM European Conference on Computer Systems (EuroSys), pages 9:1--9:14, 2014. Google ScholarDigital Library
- C.-C. Tu, M. Ferdman, C.-t. Lee, and T.-c. Chiueh. A Comprehensive Implementation and Evaluation of Direct Interrupt Delivery. In 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environment (VEE), VEE '15. ACM, 2015. Google ScholarDigital Library
- C. A. Waldspurger. Memory resource management in vmware esx server. ACM SIGOPS Operating Systems Review, 36(SI):181--194, 2002. Google ScholarDigital Library
- M. G. Xavier, M. V. Neves, F. D. Rossi, T. C. Ferreto, T. Lange, and C. A. F. De Rose. Performance Evaluation of Container-Based Virtualization for High Performance Computing Environments. In Proceedings of the 2013 21st Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP '13, pages 233--240, Washington, DC, USA, 2013. IEEE Computer Society. Google ScholarDigital Library
- B. Yee, D. Sehr, G. Dardyk, J. B. Chen, R. Muth, T. Ormandy, S. Okasaka, N. Narula, and N. Fullagar. Native client: A sandbox for portable, untrusted x86 native code. In Proceedings of the IEEE Symposium on Security and Privacy (Oakland), 2009. Google ScholarDigital Library
- Containing the Hype
Recommendations
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution EnvironmentsSingle-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
Comments