Abstract
[Context] Systems that incorporate Machine Learning (ML) models, often referred to as ML-enabled systems, have become commonplace. However, empirical evidence on how ML-enabled systems are engineered in practice is still limited; this is especially true for activities surrounding ML model dissemination. [Goal] We investigate contemporary industrial practices and problems related to ML model dissemination, focusing on the model deployment and the monitoring ML life cycle phases. [Method] We conducted an international survey to gather practitioner insights on how ML-enabled systems are engineered. We gathered a total of 188 complete responses from 25 countries. We analyze the status quo and problems reported for the model deployment and monitoring phases. We analyzed contemporary practices using bootstrapping with confidence intervals and conducted qualitative analyses on the reported problems applying open and axial coding procedures. [Results] Practitioners perceive the model deployment and monitoring phases as relevant and difficult. With respect to model deployment, models are typically deployed as separate services, with limited adoption of MLOps principles. Reported problems include difficulties in designing the architecture of the infrastructure for production deployment and legacy application integration. Concerning model monitoring, many models in production are not monitored. The main monitored aspects are inputs, outputs, and decisions. Reported problems involve the absence of monitoring practices, the need to create custom monitoring tools, and the selection of suitable metrics. [Conclusion] Our results help provide a better understanding of the adopted practices and problems in practice and support guiding ML deployment and monitoring research in a problem-driven manner.
G. Giray—Independent Researcher.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Nahar, N., Zhang, H., Lewis, G., Zhou, S., Kastner, C.: A meta-summary of challenges in building products with ml components - collecting experiences from 4758+ practitioners. In: 2023 IEEE/ACM 2nd International Conference on AI Engineering - Software Engineering for AI (CAIN), pp. 171–183. IEEE Computer Society, Los Alamitos, CA, USA, May 2023
Startech Up: Machine learning history: the complete timeline, September 2022
Paleyes, A., Urma, R.G., Lawrence, N.D.: Challenges in deploying machine learning: a survey of case studies. ACM Comput. Surv. 55(6) (2022)
John, M.M., Olsson, H.H., Bosch, J.: Towards MLOps: a framework and maturity model. In: 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 1–8 (2021)
Lewis, G.A., Ozkaya, I., Xu, X.: Software architecture challenges for ml systems. In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 634–638 (2021)
John, M.M., Olsson, H.H., Bosch, J.: AI deployment architecture: multi-case study for key factor identification. In: 2020 27th Asia-Pacific Software Engineering Conference (APSEC), pp. 395–404 (2020)
John, M.M., Holmström Olsson, H., Bosch, J.: Architecting AI deployment: a systematic review of state-of-the-art and state-of-practice literature. In: Klotins, E., Wnuk, K. (eds.) ICSOB 2020. LNBIP, vol. 407, pp. 14–29. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67292-8_2
Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)
Syafrudin, M., Alfian, G., Fitriyani, N.L., Rhee, J.: Performance analysis of IoT-based sensor, big data processing, and machine learning model for real-time monitoring system in automotive manufacturing. Sensors 18(9) (2018)
Chahal, D., Ojha, R., Ramesh, M., Singhal, R.: Migrating large deep learning models to serverless architecture, pp. 111–116 (2020). Cited by: 14
Nowrin, I., Khanam, F.: Importance of cloud deployment model and security issues of software as a service (SaaS) for cloud computing. In: 2019 International Conference on Applied Machine Learning (ICAML), pp. 183–186 (2019)
Mrozek, D., Koczur, A., Małysiak-Mrozek, B.: Fall detection in older adults with mobile IoT devices and machine learning in the cloud and on the edge. Inf. Sci. 537, 132–147 (2020). Cited by: 72. All Open Access, Hybrid Gold Open Access (2020)
Abdelaziz, A., Elhoseny, M., Salama, A.S., Riad, A.: A machine learning model for improving healthcare services on cloud computing environment. Measurement 119, 117–128 (2018)
Garg, S., Pundir, P., Rathee, G., Gupta, P., Garg, S., Ahlawat, S.: On continuous integration/continuous delivery for automated deployment of machine learning models using MLOps. In: 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pp. 25–28 (2021)
Al-Doghman, F., Moustafa, N., Khalil, I., Sohrabi, N., Tari, Z., Zomaya, A.Y.: AI-enabled secure microservices in edge computing: opportunities and challenges. IEEE Trans. Serv. Comput. 16(2), 1485–1504 (2023)
Paraskevoulakou, E., Kyriazis, D.: ML-FaaS: towards exploiting the serverless paradigm to facilitate machine learning functions as a service. IEEE Trans. Netw. Serv. Manag. 20, 2110–2123 (2023)
Kourouklidis, P., Kolovos, D., Noppen, J., Matragkas, N.: A model-driven engineering approach for monitoring machine learning models. In: 2021 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pp. 160–164 (2021)
Schröder, T., Schulz, M.: Monitoring machine learning models: a categorization of challenges and methods. Data Sci. Manag. 5(3), 105–116 (2022)
Wagner, S., Mendez, D., Felderer, M., Graziotin, D., Kalinowski, M.: Challenges in survey research. In: Contemporary Empirical Methods in Software Engineering, pp. 93–125. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-32489-6_4
Lunneborg, C.E.: Bootstrap inference for local populations. Ther. Innov. Regul. Sci. 35(4), 1327–1342 (2001)
Wagner, S., et al.: Status quo in requirements engineering: a theory and a global family of surveys. ACM Trans. Softw. Eng. Methodol. 28(2) (2019)
Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman & Hall/CRC (1993)
Lei, S., Smith, M.: Evaluation of several nonparametric bootstrap methods to estimate confidence intervals for software metrics. IEEE Trans. Software Eng. 29(11), 996–1004 (2003)
Stol, K.J., Ralph, P., Fitzgerald, B.: Grounded theory in software engineering research: a critical review and guidelines. In: Proceedings of the 38th International Conference on Software Engineering, pp. 120–131 (2016)
Kalinowski, M., Escovedo, T., Villamizar, H., Lopes, H.: Engenharia de Software para Ciência de Dados: Um guia de boas práticas com ênfase na construção de sistemas de Machine Learning em Python. Casa do Código (2023)
Amershi, S., et al.: Software engineering for machine learning: a case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, pp. 291–300. IEEE (2019)
Schröer, C., Kruse, F., Gómez, J.M.: A systematic literature review on applying CRISP-DM process model. Procedia Comput. Sci. 181, 526–534 (2021)
GitLab: Get started with GitLab CI/CD, October 2023
Azure DevOps: What is Azure DevOps? October 2022
BentoML: What is bentoml? October 2023
MLflow: What is mlflow? October 2023
AWS: Amazon SageMaker for MLOPs, October 2023
Kalinowski, M., Mendes, E., Card, D.N., Travassos, G.H.: Applying DPPI: a defect causal analysis approach using Bayesian networks. In: Ali Babar, M., Vierimaa, M., Oivo, M. (eds.) PROFES 2010. LNCS, vol. 6156, pp. 92–106. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13792-1_9
Kalinowski, M., Mendes, E., Travassos, G.H.: Automating and evaluating probabilistic cause-effect diagrams to improve defect causal analysis. In: Caivano, D., Oivo, M., Baldassarre, M.T., Visaggio, G. (eds.) PROFES 2011. LNCS, vol. 6759, pp. 232–246. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21843-9_19
Mäkinen, S., Skogström, H., Laaksonen, E., Mikkonen, T.: Who needs MLOps: what data scientists seek to accomplish and how can MLOps help? In: 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN), pp. 109–112 (2021)
Ruf, P., Madan, M., Reich, C., Ould-Abdeslam, D.: Demystifying MLOps and presenting a recipe for the selection of open-source tools. Appl. Sci. 11(19) (2021)
Zhou, Y., Yu, Y., Ding, B.: Towards MLOps: a case study of ML pipeline platform. In: 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE), pp. 494–500 (2020)
Algorithmia: 2020 state of enterprise machine learning. Technical report (2019)
Siegel, E.: Models are rarely deployed: an industry-wide failure in machine learning leadership, January 2022
Weiner, J.: Why AI/Data Science Projects Fail: How to Avoid Project Pitfalls. Claypool Publishers, Morgan (2021)
Heymann, H., Kies, A.D., Frye, M., Schmitt, R.H., Boza, A.: Guideline for deployment of machine learning models for predictive quality in production. Procedia CIRP 107, 815–820 (2022). Leading Manufacturing Systems Transformation - Proceedings of the 55th CIRP Conference on Manufacturing Systems (2022)
Linaker, J., Sulaman, S.M., Höst, M., de Mello, R.M.: Guidelines for conducting surveys in software engineering v. 1.1. Lund University 50 (2015)
Fernández, D.M., et al.: Naming the pain in requirements engineering: contemporary problems, causes, and effects in practice. Empir. Softw. Eng. 22, 2298–2338 (2017)
Kim, M., Zimmermann, T., DeLine, R., Begel, A.: Data scientists in software teams: state of the art and challenges. IEEE Trans. Software Eng. 44(11), 1024–1038 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zimelewicz, E. et al. (2024). ML-Enabled Systems Model Deployment and Monitoring: Status Quo and Problems. In: Bludau, P., Ramler, R., Winkler, D., Bergsmann, J. (eds) Software Quality as a Foundation for Security. SWQD 2024. Lecture Notes in Business Information Processing, vol 505. Springer, Cham. https://doi.org/10.1007/978-3-031-56281-5_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-56281-5_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56280-8
Online ISBN: 978-3-031-56281-5
eBook Packages: Computer ScienceComputer Science (R0)