ABSTRACT
Machine learning (ML) components are being added to more and more critical and impactful software systems, but the software development process of real-world production systems from prototyped ML models remains challenging with additional complexity and interdisciplinary collaboration challenges. This poses difficulties in using traditional software lifecycle models such as waterfall, spiral, or agile models when building ML-enabled systems. In this research, we apply a Systems Engineering lens to investigate the use of V-Model in addressing the interdisciplinary collaboration challenges when building ML-enabled systems. By interviewing practitioners from software companies, we established a set of 8 propositions for using V-Model to manage interdisciplinary collaborations when building products with ML components. Based on the propositions, we found that despite requiring additional efforts, the characteristics of V-Model align effectively with several collaboration challenges encountered by practitioners when building ML-enabled systems. We recommend future research to investigate new process models that leverage the characteristics of V-Model such as the system decomposition, clear system boundary, and consistency of Validation & Verification (V&V) for building ML-enabled systems.
- Rama Akkiraju, Vibha Sinha, Anbang Xu, Jalal Mahmud, Pritam Gundecha, Zhe Liu, Xiaotong Liu, and John Schumacher. 2020. Characterizing machine learning processes: A maturity framework. In Business Process Management: 18th International Conference, BPM 2020, Seville, Spain, September 13--18, 2020, Proceedings 18. Springer, 17--31.Google Scholar
Digital Library
- Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 291--300.Google ScholarDigital Library
- Anders Arpteg, Björn Brinne, Luka Crnkovic-Friis, and Jan Bosch. 2018. Software engineering challenges of deep learning. In 2018 44th euromicro conference on software engineering and advanced applications (SEAA). IEEE, 50--59.Google Scholar
- Sundramoorthy Balaji and M Sundararajan Murugaiyan. 2012. Waterfall vs. V-Model vs. Agile: A comparative study on SDLC. International Journal of Information Technology and Business Management 2, 1 (2012), 26--30.Google Scholar
- David Beale and Joseph Bonometti. 2006. Systems engineering (SE)-the systems design process. The Lunar Engineering Handbook, Auburg University, Auburn (2006).Google Scholar
- Thomas F Bersson, Thomas Mazzuchi, and Shahram Sarkani. 2012. A framework for application of system engineering process models to sustainable design of high performance buildings. Journal of Green Building 7, 3 (2012), 171--192.Google ScholarCross Ref
- Alex Bitektine. 2008. Prospective case study design: Qualitative method for deductive theory testing. Organizational research methods 11, 1 (2008), 160--180.Google ScholarCross Ref
- Benjamin S. Blanchard and Wolter J. Fabrycky. 2011. Systems Engineering and Analysis. Prentice Hall.Google Scholar
- DA Bodner and WB Rouse. 2009. Handbook of systems engineering and management. Wiley. chapter Organizational Simulation (2009).Google Scholar
- Markus Borg, Cristofer Englund, Krzysztof Wnuk, Boris Duran, Christoffer Levandowski, Shenjian Gao, Yanwen Tan, Henrik Kaijser, Henrik Lönn, and Jonas Törnqvist. 2019. Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry. Journal of Automotive Software Engineering 1, 1 (2019), 1--19.Google ScholarCross Ref
- Jan Bosch, Helena Holmström Olsson, and Ivica Crnkovic. 2021. Engineering ai systems: A research agenda. Artificial Intelligence Paradigms for Smart Cyber-Physical Systems (2021), 1--19.Google Scholar
- Houssem Ben Braiek and Foutse Khomh. 2020. On testing machine learning programs. Journal of Systems and Software 164 (2020), 110542.Google ScholarCross Ref
- Eric J Braude and Michael E Bernstein. 2016. Software engineering: modern approaches. Waveland Press.Google Scholar
- Adolf-Peter Bröhl. 1993. Das V-Modell: Der Standard für die Softwareentwicklung mit Praxisleitfaden. Oldenbourg.Google Scholar
- F Cechini, R Ice, and D Binkley. 2009. Systems Engineering Guidebook for Intelligent Transportation Systems. California Division of the United States Department of Transportation Federal Highway Administration and the California Department of Transportation (2009).Google Scholar
- Johan Cederbladh, Antonio Cicchetti, and Jagadish Suryadevara. 2023. Early Validation and Verification of System Behaviour in Model-Based Systems Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology (2023).Google Scholar
- Jiyoo Chang and Christine Custis. 2022. Understanding Implementation Challenges in Machine Learning Documentation. In Equity and Access in Algorithms, Mechanisms, and Optimization. 1--8.Google Scholar
- Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX design innovation: Challenges for working with machine learning as a design material. In Proceedings of the 2017 chi conference on human factors in computing systems. 278--288.Google ScholarDigital Library
- Gary Ericson, William Anton Rohm, Josée Martens, Kent Sharkey, Craig Casey, Beth Harvey, and Nick Schonning. 2017. Team data science process documentation. Retrieved April 11 (2017), 2019.Google Scholar
- James Fanson. 2010. Lessons learned from the Kepler Mission and space telescope management. In An Optical Believe It or Not: Key Lessons Learned II, Vol. 7796. SPIE, 25--30.Google Scholar
- Daniel R Georgiadis, Thomas A Mazzuchi, and Shahram Sarkani. 2013. Using multi criteria decision making in analysis of alternatives for selection of enabling technology. Systems Engineering 16, 3 (2013), 287--303.Google ScholarCross Ref
- Görkem Giray. 2021. A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software 180 (2021), 111031.Google ScholarDigital Library
- Valentina Golendukhina, Valentina Lenarduzzi, and Michael Felderer. 2022. What is software quality for AI engineers? Towards a thinning of the fog. In Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI. 1--9.Google ScholarDigital Library
- Iris Graessler and Julian Hentze. 2020. The new V-Model of VDI 2206 and its validation. at-Automatisierungstechnik 68, 5 (2020), 312--324.Google Scholar
- Venkat Gudivada, Amy Apon, and Junhua Ding. 2017. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. International Journal on Advances in Software 10, 1 (2017), 1--20.Google Scholar
- Mark Haakman, Luis Cruz, Hennie Huijgens, and Arie van Deursen. 2021. AI lifecycle models need to be revised: An exploratory study in Fintech. Empirical Software Engineering 26 (2021), 1--29.Google ScholarDigital Library
- Reinhard Haberfellner, Olivier De Weck, Ernst Fricke, and Siegfried Vössner. 2019. Systems engineering: fundamentals and applications. Springer.Google Scholar
- Ling Huang, Anthony D. Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J. Doug Tygar. 2011. Adversarial machine learning. In Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence. 43--58.Google Scholar
- Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. 2020. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review 37 (2020), 100270.Google ScholarCross Ref
- Meenu Mary John, Helena Holmström Olsson, and Jan Bosch. 2020. Ai deployment architecture: Multi-case study for key factor identification. In 2020 27th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 395--404.Google ScholarCross Ref
- Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2017. Data scientists in software teams: State of the art and challenges. IEEE Transactions on Software Engineering 44, 11 (2017), 1024--1038.Google ScholarCross Ref
- Alexander Kossiakoff, William N. Sweet, Sam Seymour, and Steven M. Biemer. 2011. Systems Engineering: Principles and Practice. John Wiley & Sons.Google Scholar
- Samuli Laato, Matti Mäntymäki, Matti Minkkinen, Teemu Birkstedt, AKM Islam, and Denis Dennehy. 2022. Integrating machine learning with software development lifecycles: Insights from experts. (2022).Google Scholar
- Grace A Lewis, Ipek Ozkaya, and Xiwei Xu. 2021. Software architecture challenges for ml systems. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 634--638.Google ScholarCross Ref
- Shuyue Li, Jiaqi Guo, Jian-Guang Lou, Ming Fan, Ting Liu, and Dongmei Zhang. 2022. Testing machine learning systems in industry: an empirical study. In Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice. 263--272.Google ScholarDigital Library
- Hanyan Liu, Samuel Eksmo, Johan Risberg, and Regina Hebig. 2020. Emerging and changing tasks in the development process for machine learning systems. In Proceedings of the international conference on software and system processes. 125--134.Google ScholarDigital Library
- Qiang Liu, Pan Li, Wentao Zhao, Wei Cai, Shui Yu, and Victor CM Leung. 2018. A survey on security threats and defensive techniques of machine learning: A data driven view. IEEE access 6 (2018), 12103--12117.Google Scholar
- Mark W. Maier and Eberhardt Rechtin. 2009. The Art of Systems Architecting. CRC Press.Google Scholar
- Sasu Mäkinen, Henrik Skogström, Eero Laaksonen, and Tommi Mikkonen. 2021. Who needs MLOps: What data scientists seek to accomplish and how can MLOps help?. In 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN). IEEE, 109--112.Google ScholarCross Ref
- Fernando Martínez-Plumed, Lidia Contreras-Ochando, Cesar Ferri, José Hernández-Orallo, Meelis Kull, Nicolas Lachiche, Maria José Ramírez-Quíntana, and Peter Flach. 2019. CRISP-DM twenty years later: From data mining processes to data science trajectories. IEEE Transactions on Knowledge and Data Engineering 33, 8 (2019), 3048--3061.Google ScholarCross Ref
- John TJ Mathieson, Thomas Mazzuchi, and Shahram Sarkani. 2020. The systems engineering DevOps lemniscate and model-based system operations. IEEE Systems Journal 15, 3 (2020), 3980--3991.Google ScholarCross Ref
- Gary McGraw, Harold Figueroa, Victor Shepardson, and Richie Bonett. 2020. An architectural risk analysis of machine learning systems: Toward more secure machine learning. Technical Report. Berryville Institute of Machine Learning, v 1.0.Google Scholar
- Naja Holten Møller, Claus Bossen, Kathleen H. Pine, Trine Rask Nielsen, and Gina Neff. 2020. Who does the work of data? Interactions 27, 3 (2020), 52--55.Google ScholarDigital Library
- Nadia Nahar, Haoran Zhang, Grace Lewis, Shurui Zhou, and Christian Kästner. 2023. A Meta-Summary of Challenges in Building Products with ML Components-Collecting Experiences from 4758+ Practitioners. arXiv preprint arXiv:2304.00078 (2023).Google Scholar
- Nadia Nahar, Shurui Zhou, Grace Lewis, and Christian Kästner. 2022. Collaboration challenges in building ml-enabled systems: Communication, documentation, engineering, and process. In Proceedings of the 44th International Conference on Software Engineering. 413--425.Google ScholarDigital Library
- Morteza Namvar, Ali Intezari, Saeed Akhlaghpour, and Justin P Brienza. 2023. Beyond effective use: Integrating wise reasoning in machine learning development. International Journal of Information Management 69 (2023), 102566.Google ScholarDigital Library
- Paul R Niven and Ben Lamorte. 2016. Objectives and key results: Driving focus, alignment, and engagement with OKRs. John Wiley & Sons.Google Scholar
- Katie O'Leary and Makoto Uchida. 2020. Common problems with creating machine learning pipelines from existing code. (2020).Google Scholar
- Ipek Ozkaya. 2020. What is really different in engineering AI-enabled systems? IEEE software 37, 4 (2020), 3--6.Google Scholar
- Patrick Petersen, Hanno Stage, Jacob Langner, Lennart Ries, Philipp Rigoll, Carl Philipp Hohl, and Eric Sax. 2022. Towards a Data Engineering Process in Data-Driven Systems Engineering. In 2022 IEEE International Symposium on Systems Engineering (ISSE). IEEE, 1--8.Google ScholarCross Ref
- Romesh Ranawana and Asoka S Karunananda. 2021. An agile software development life cycle model for machine learning application development. In 2021 5th SLAAI International Conference on Artificial Intelligence (SLAAI-ICAI). IEEE, 1--6.Google ScholarCross Ref
- Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing machine learning based systems: a systematic mapping. Empirical Software Engineering (2020), 1--62.Google Scholar
- Jane Ritchie and Liz Spencer. 2002. Qualitative data analysis for applied policy research. In Analyzing qualitative data. Routledge, 173--194.Google Scholar
- Andrew P Sage and William B Rouse. 2014. Handbook of systems engineering and management. John Wiley & Sons.Google Scholar
- Rick Salay and Krzysztof Czarnecki. 2018. Using machine learning safely in automotive software: An assessment and adaption of software process requirements in ISO 26262. arXiv preprint arXiv:1808.01614 (2018).Google Scholar
- Rick Salay, Rodrigo Queiroz, and Krzysztof Czarnecki. 2017. An analysis of ISO 26262: Using machine learning safely in automotive software. arXiv preprint arXiv:1709.02435 (2017).Google Scholar
- Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M. Aroyo. 2021. "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI. In Proceedings of the Conference on Human Factors in Computing Systems. 1--15.Google Scholar
- Nithya Sambasivan and Rajesh Veeraraghavan. 2022. The Deskilling of Domain Expertise in AI Development. In Proceedings of the Conference on Human Factors in Computing Systems. 1--14.Google ScholarDigital Library
- Iqbal H Sarker. 2021. Machine learning: Algorithms, real-world applications and research directions. SN computer science 2, 3 (2021), 160.Google Scholar
- Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, and Andreas Grafberger. 2018. Automating Large-Scale Data Quality Verification. Proceedings of the VLDB Endowment 11, 12 (2018), 1781--1794.Google ScholarDigital Library
- David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in neural information processing systems 28 (2015).Google Scholar
- Alex Serban, Koen van der Blom, Holger Hoos, and Joost Visser. 2020. Adoption and effects of software engineering best practices in machine learning. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1--12.Google Scholar
- Alex Serban and Joost Visser. 2022. Adapting software architectures to machine learning challenges. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 152--163.Google ScholarCross Ref
- Mary Shaw and Liming Zhu. 2022. Can software engineering harness the benefits of advanced AI? IEEE Software 39, 6 (2022), 99--104.Google ScholarDigital Library
- Md Saeed Siddik and Cor-Paul Bezemer. 2023. Do Code Quality and Style Issues Differ Across (Non-) Machine Learning Notebooks? Yes!. In 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 72--83.Google ScholarCross Ref
- Micah J Smith, Carles Sala, James Max Kanter, and Kalyan Veeramachaneni. 2020. The machine learning bazaar: Harnessing the ml ecosystem for effective system development. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 785--800.Google ScholarDigital Library
- Stefan Studer, Thanh Binh Bui, Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters, and Klaus-Robert Müller. 2021. Towards CRISP-ML (Q): a machine learning process model with quality assurance methodology. Machine learning and knowledge extraction 3, 2 (2021), 392--413.Google Scholar
- Andreas Vogelsang and Markus Borg. 2019. Requirements engineering for machine learning: Perspectives from data scientists. In 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW). IEEE, 245--251.Google ScholarCross Ref
- David D Walden et al. 2015. Systems engineering handbook: A guide for system life cycle processes and activities. (2015).Google Scholar
- Zhiyuan Wan, Xin Xia, David Lo, and Gail C Murphy. 2019. How does machine learning change software development practices? IEEE Transactions on Software Engineering 47, 9 (2019), 1857--1871.Google Scholar
- Charles S. Wasson. 2006. Systems Engineering: Coping with Complexity. John Wiley & Sons.Google Scholar
- Steven Euijong Whang and Jae-Gil Lee. 2020. Data collection and quality challenges for deep learning. Proceedings of the VLDB Endowment 13, 12 (2020), 3429--3432.Google ScholarDigital Library
- Wikipedia. 2023. Systems engineering --- Wikipedia, The Free Encyclopedia. http://en.wikipedia.org/w/index.php?title=Systems%20engineering&oldid=1163252030. [Online; accessed 08-August-2023].Google Scholar
- Carl Wilhjelm and Awad A. Younis. 2020. A threat analysis methodology for security requirements elicitation in machine learning based systems. In 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 426--433.Google Scholar
- Jie JW Wu, Thomas A Mazzuchi, and Shahram Sarkani. 2023. Comparison of multi-criteria decision-making methods for online controlled experiments in a launch decision-making framework. Information and Software Technology 155 (2023), 107115.Google ScholarDigital Library
- Oliver Zendel, Markus Murschitz, Martin Humenberger, and Wolfgang Herzner. 2015. CV-HAZOP: Introducing test data validation for computer vision. In Proceedings of the IEEE International Conference on Computer Vision. 2066--2074.Google ScholarDigital Library
- Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering (2020).Google ScholarDigital Library
Recommendations
An Agile Perspective on Open Source Software Engineering
Open source software OSS development has been a trend parallel to that of agile software development, which is the highly iterative development model following conventional software engineering principles. Striking similarities exist between the two ...
Teaching software engineering practices with Extreme Programming
Extreme Programming (XP), one of many models for software development, has challenged some traditional software engineering practices while taking others to the extreme. The controversial practices raise questions about the role of XP in teaching ...
Comments