Hojin Park
George Amvrosiadis
ABSTRACT
Public cloud providers offer a diverse collection of storage types and configurations with different costs and performance SLAs. As a consequence, it is difficult to select the most cost-efficient allocations for storage backends, while satisfying a given workload's performance requirements, when moving data-heavy applications to the cloud. We present Mimir, a tool for automatically finding a cost-efficient virtual storage cluster configuration for a customer's storage workload and performance requirements. Importantly, Mimir considers all block storage types and configurations, and even heterogeneous mixes of them. In our experiments, compared to state-of-the-art approaches that consider only one storage type, Mimir finds configurations that reduce cost by up to 81% for real-application-based key-value store workloads.
- Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. 2017. CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 469--482. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/alipourfardGoogle Scholar
Digital Library
- Guillermo A. Alvarez, Elizabeth Borowsky, Susie Go, Theodore H. Romer, Ralph A. Becker-Szendy, Richard A. Golding, Arif Merchant, Mirjana Spasojevic, Alistair C. Veitch, and John Wilkes. 2001. Minerva: An automated resource provisioning tool for large-scale storage systems. ACM Trans. Comput. Syst. 19 (2001), 483--518.Google ScholarDigital Library
- Eric Anderson, Michael Hobbs, Kimberly Keeton, Susan Spence, Mustafa Uysal, and Alistair Veitch. 2002. Hippodrome: Running Circles Around Storage Administration. In Conference on File and Storage Technologies (FAST 02). USENIX Association, Monterey, CA. https://www.usenix.org/conference/fast-02/hippodrome-running-circles-around-storage-administrationGoogle Scholar
- George E. Andrews. 1976. The Theory of Partitions. Cambridge University Press.Google Scholar
- Berk Atikoglu, Yuehai Xu, Eitan Frachtenberg, Song Jiang, and Mike Paleczny. 2012. Workload Analysis of a Large-Scale Key-Value Store. In ACM SIGMET-RICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS '12, London, United Kingdom, June 11-15, 2012 (London, England, UK) (SIGMETRICS '12). Association for Computing Machinery, New York, NY, USA, 53--64. Google ScholarDigital Library
- Jens Axboe. 2022. Flexible I/O Tester. https://github.com/axboe/fioGoogle Scholar
- Vasanth Balasundaram, Geoffrey Fox, Ken Kennedy, and Ulrich Kremer. 1991. A Static Performance Estimator to Guide Data Partitioning Decisions. In Proceedings of the Third ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (Williamsburg, Virginia, USA) (PPOPP '91). Association for Computing Machinery, New York, NY, USA, 213--223. Google ScholarDigital Library
- Muhammad Bilal, Marco Canini, and Rodrigo Rodrigues. 2020. Finding the Right Cloud Configuration for Analytics Clusters. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA) (SoCC '20). Association for Computing Machinery, New York, NY, USA, 208--222. Google ScholarDigital Library
- Zhichao Cao, Siying Dong, Sagar Vemuri, and David H.C. Du. 2020. Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook. In 18th USENIX Conference on File and Storage Technologies (FAST 20). USENIX Association, Santa Clara, CA, 209--223. https://www.usenix.org/conference/fast20/presentation/cao-zhichaoGoogle Scholar
- Surajit Chaudhuri, Vivek Narasayya, and Ravishankar Ramamurthy. 2004. Estimating progress of execution for SQL queries (SIGMOD '04). Association for Computing Machinery, New York, NY, USA, 803--814. Google ScholarDigital Library
- Yue Cheng, Aayush Gupta, and Ali Raza Butt. 2015. An in-memory object caching framework with adaptive load balancing. In Proceedings of the Tenth European Conference on Computer Systems, EuroSys 2015, Bordeaux, France, April 21-24, 2015, Laurent Réveillère, Tim Harris, and Maurice Herlihy (Eds.). ACM, 4:1--4:16. Google ScholarDigital Library
- Andrew Chung, Jun Woo Park, and Gregory R. Ganger. 2018. Stratus: cost-aware container scheduling in the public cloud. In Proceedings of the ACM Symposium on Cloud Computing (Carlsbad, CA, USA) (SoCC '18). Association for Computing Machinery, New York, NY, USA, 121--134. Google ScholarDigital Library
- Christina Delimitrou and Christos Kozyrakis. 2014. Quasar: Resource-Efficient and QoS-Aware Cluster Management. SIGPLAN Not. 49, 4 (Feb. 2014), 127--144. Google ScholarDigital Library
- Salvatore Dipietro, Giuliano Casale, and Giuseppe Serazzi. 2017. A Queueing Network Model for Performance Prediction of Apache Cassandra. In Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools on 10th EAI International Conference on Performance Evaluation Methodologies and Tools (Taormina, Italy) (VALUETOOLS'16). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), Brussels, BEL, 186--193. Google ScholarDigital Library
- Jerome Friedman. 2001. Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics 29 (10 2001), 1189--1232. Google ScholarCross Ref
- Daniel Gmach, Jerry Rolia, Ludmila Cherkasova, and Alfons Kemper. 2007. Workload Analysis and Demand Prediction of Enterprise Data Center Applications. In IEEE 10th International Symposium on Workload Characterization, IISWC 2007, Boston, MA, USA, 27-29 September, 2007. IEEE Computer Society, 171--180. Google ScholarDigital Library
- Google. 2022. Google Compute Engine Persistent Disks. https://cloud.google.com/compute/docs/disksGoogle Scholar
- Gurobi Optimization, LLC. 2021. Gurobi Optimizer Reference Manual. https://www.gurobi.comGoogle Scholar
- Jonathan R. M. Hosking and Jamie Wallis. 1987. Parameter and quantile estimation for the generalized pareto distribution. Technometrics 29 (1987), 339--349.Google ScholarDigital Library
- Dongxu Huang, Qi Liu, Qiu Cui, Zhuhe Fang, Xiaoyu Ma, Fei Xu, Ling Shen, Liu Tang, Yuxing Zhou, Menglong Huang, Wan Wei, Cong Liu, Jian Zhang, Jianjun Li, Xuelian Wu, Lingyu Song, Ruoxi Sun, Shuaipeng Yu, Lei Zhao, Nicholas Cameron, Liquan Pei, and Xin Tang. 2020. TiDB: A Raft-based HTAP Database. Proc. VLDB Endow. 13 (2020), 3072--3084.Google ScholarDigital Library
- Flavio P. Junqueira, Ivan Kelly, and Benjamin Reed. 2013. Durability with BookKeeper. SIGOPS Oper. Syst. Rev. 47, 1 (Jan. 2013), 9--15. Google ScholarDigital Library
- Arijit Khan, Xifeng Yan, Shu Tao, and Nikos Anerousis. 2012. Workload characterization and prediction in the cloud: A multiple time series approach. In 2012 IEEE Network Operations and Management Symposium, NOMS 2012, Maui, HI, USA, April 16-20, 2012, Filip De Turck, Luciano Paschoal Gaspary, and Deep Medhi (Eds.). IEEE, 1287--1294. Google ScholarCross Ref
- Markus Klems, Adam Silberstein, Jianjun Chen, Masood Mortazavi, Sahaya Andrews Albert, P. P. S. Narayan, Adwait Tumbde, and Brian F. Cooper. 2012. The Yahoo!: cloud datastore load balancer. In Proceedings of the Fourth International Workshop on Cloud Data Management, CloudDB 2012, Maui, HI, USA, October 29, 2012, Xiaofeng Meng, Adam Silberstein, and Fusheng Wang (Eds.). ACM, 33--40. Google ScholarDigital Library
- Ana Klimovic, Heiner Litz, and Christos Kozyrakis. 2018. Selecta: Heterogeneous Cloud Storage Configuration for Data Analytics. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 759--773. https://www.usenix.org/conference/atc18/presentation/klimovic-selectaGoogle Scholar
- Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos Kozyrakis. 2018. Pocket: Elastic Ephemeral Storage for Serverless Analytics. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 427--444. https://www.usenix.org/conference/osdi18/presentation/klimovicGoogle ScholarDigital Library
- Chunbo Lai, Song Jiang, Liqiong Yang, Shiding Lin, Guangyu Sun, Zhenyu Hou, Can Cui, and Jason Cong. 2015. Atlas: Baidu's key-value storage system for cloud data. In IEEE 31st Symposium on Mass Storage Systems and Technologies, MSST 2015, Santa Clara, CA, USA, May 30 - June 5, 2015. IEEE Computer Society, 1--14. Google ScholarCross Ref
- Viktor Leis and Maximilian Kuschewski. 2021. Towards Cost-Optimal Query Processing in the Cloud. Proc. VLDB Endow. 14 (2021), 1606--1612.Google ScholarDigital Library
- Yongkun Li, Zhen Liu, Patrick P. C. Lee, Jiayu Wu, Yinlong Xu, Yi Wu, Liu Tang, Qi Liu, and Qiu Cui. 2021. Differentiated Key-Value Storage Management for Balanced I/O Performance. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 673--687. https://www.usenix.org/conference/atc21/presentation/li-yongkunGoogle Scholar
- Zaoxing Liu, Zhihao Bai, Zhenming Liu, Xiaozhou Li, Changhoon Kim, Vladimir Braverman, Xin Jin, and Ion Stoica. 2019. DistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed Caching. In 17th USENIX Conference on File and Storage Technologies, FAST 2019, Boston, MA, February 25-28, 2019, Arif Merchant and Hakim Weatherspoon (Eds.). USENIX Association, 143--157. https://www.usenix.org/conference/fast19/presentation/liuGoogle Scholar
- Ashraf Mahgoub, Alexander Michaelson Medoff, Rakesh Kumar, Subrata Mitra, Ana Klimovic, Somali Chaterji, and Saurabh Bagchi. 2020. OPTIMUSCLOUD: Heterogeneous Configuration Optimization for Distributed Databases in the Cloud. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, USA, 189--203. https://www.usenix.org/conference/atc20/presentation/mahgoubGoogle Scholar
- Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh. 2019. Learning scheduling algorithms for data processing clusters. In Proceedings of the ACM Special Interest Group on Data Communication, SIGCOMM 2019, Beijing, China, August 19-23, 2019, Jianping Wu and Wendy Hall (Eds.). ACM, 270--288. Google ScholarDigital Library
- Microsoft. 2022. Azure Disk Storage. https://azure.microsoft.com/en-us/services/storage/disksGoogle Scholar
- Subrata Mitra, Shanka Subhra Mondal, Nikhil Sheoran, Neeraj Dhake, Ravinder Nehra, and Ramanuja Simha. 2019. DeepPlace: Learning to Place Applications in Multi-Tenant Clusters. In Proceedings of the 10th ACM SIGOPS Asia-Pacific Workshop on Systems, APSys 2019, Hangzhou, China, Augsut 19-20, 2019. ACM, 61--68. Google ScholarDigital Library
- Kristi Morton, Magdalena Balazinska, and Dan Grossman. 2010. ParaTimer: A progress indicator for MapReduce DAGs. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2010, Indianapolis, Indiana, USA, June 6-10, 2010, Ahmed K. Elmagarmid and Divyakant Agrawal (Eds.). ACM, 507--518. Google ScholarDigital Library
- Barzan Mozafari, Carlo Curino, and Samuel Madden. 2013. DBSeer: Resource and Performance Prediction for Building a Next Generation Database Cloud. In Sixth Biennial Conference on Innovative Data Systems Research, CIDR 2013, Asilomar, CA, USA, January 6-9, 2013, Online Proceedings. www.cidrdb.org. http://cidrdb.org/cidr2013/Papers/CIDR13_Paper52.pdfGoogle Scholar
- Oliver Niehörster, Alexander Krieger, Jens Simon, and André Brinkmann. 2011. Autonomic Resource Management with Support Vector Machines. In 12th IEEE/ACM International Conference on Grid Computing, GRID 2011, Lyon, France, September 21-23, 2011, Shantenu Jha, Nils gentschen Felde, Rajkumar Buyya, and Gilles Fedak (Eds.). IEEE Computer Society, 157--164. Google ScholarDigital Library
- Andrew Or, Haoyu Zhang, and Michael J. Freedman. 2020. Resource Elasticity in Distributed Deep Learning. In MLSys. mlsys.org, USA.Google Scholar
- Chenhao Qu, Rodrigo N. Calheiros, and Rajkumar Buyya. 2018. Auto-Scaling Web Applications in Clouds: A Taxonomy and Survey. ACM Comput. Surv. 51, 4, Article 73 (July 2018), 33 pages. Google ScholarDigital Library
- Amazon Web Services. 2022. Amazon EBS volume types. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#hard-disk-drivesGoogle Scholar
- Amazon Web Services. 2022. Amazon Elastic Block Store. https://aws.amazon.com/ebsGoogle Scholar
- Supreeth Shastri and David E. Irwin. 2017. HotSpot: automated server hopping in cloud spot markets. In Proceedings of the 2017 Symposium on Cloud Computing, SoCC 2017, Santa Clara, CA, USA, September 24-27, 2017. ACM, USA, 493--505. Google ScholarDigital Library
- Manish Shukla and Sanjay Jharkharia. 2013. Applicability of ARIMA Models in Wholesale Vegetable Market: An Investigation. Int. J. Inf. Syst. Supply Chain Manag. 6, 3 (jul 2013), 105--119. Google ScholarDigital Library
- John D. Strunk, Eno Thereska, Christos Faloutsos, and Gregory R. Ganger. 2008. Using Utility to Provision Storage Systems. In 6th USENIX Conference on File and Storage Technologies, FAST 2008, February 26-29, 2008, San Jose, CA, USA, Mary Baker and Erik Riedel (Eds.). USENIX, USA, 313--328. http://www.usenix.org/events/fast08/tech/strunk.htmlGoogle Scholar
- Shivaram Venkataraman, Zongheng Yang, Michael Franklin, Benjamin Recht, and Ion Stoica. 2016. Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics. In 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16). USENIX Association, Santa Clara, CA, 363--378. https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/venkataramanGoogle ScholarDigital Library
- Haoyu Wang, Haiying Shen, Qi Liu, Kevin Zheng, and Jie Xu. 2020. A Reinforcement Learning Based System for Minimizing Cloud Storage Service Cost. In 49th International Conference on Parallel Processing - ICPP (Edmonton, AB, Canada) (ICPP '20). Association for Computing Machinery, New York, NY, USA, Article 30, 10 pages. Google ScholarDigital Library
- Peng Wang, Haixun Wang, and Wei Wang. 2011. Finding semantics in time series. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2011, Athens, Greece, June 12-16, 2011, Timos K. Sellis, Renée J. Miller, Anastasios Kementsietsidis, and Yannis Velegrakis (Eds.). ACM, 385--396. Google ScholarDigital Library
- Neeraja J. Yadwadkar, Bharath Hariharan, Joseph E. Gonzalez, Burton Smith, and Randy H. Katz. 2017. Selecting the Best VM across Multiple Public Clouds: A Data-Driven Performance Modeling Approach. In Proceedings of the 2017 Symposium on Cloud Computing (Santa Clara, California) (SoCC '17). Association for Computing Machinery, New York, NY, USA, 452--465. Google ScholarDigital Library
- Peipei Zhou, Jiayi Sheng, Cody Hao Yu, Peng Wei, Jie Wang, Di Wu, and Jason Cong. 2021. MOCHA: Multinode Cost Optimization in Heterogeneous Clouds with Accelerators. In The 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (Virtual Event, USA) (FPGA '21). Association for Computing Machinery, New York, NY, USA, 273--279. Google ScholarDigital Library
Index Terms
- Mimir: Finding Cost-efficient Storage Configurations in the Public Cloud
Recommendations
A public cloud storage auditing scheme for resource-constrained clients
Cloud storage is becoming more and more popular with the development of cloud computing technique. Cloud storage auditing, as one important security service, is used to check the integrity of the cloud data stored in cloud for users. However, the burden ...
Optimal resource provisioning for cloud computing environment
The paper presents an efficient cloud resource provisioning approach. The Software as a Service (SaaS) provider leases resources from cloud providers and also leases software as services to SaaS users. The SaaS providers aim at minimizing the payment of ...
Optimization of Resource Provisioning Cost in Cloud Computing
In cloud computing, cloud providers can offer cloud consumers two provisioning plans for computing resources, namely reservation and on-demand plans. In general, cost of utilizing computing resources provisioned by reservation plan is cheaper than that ...
Comments