Rogério de Lemos
ABSTRACT
The main goal of any feedback control system is essentially to remove humans from the loop. This has always been the goal in the engineering of control systems. The MAPE-K loop is the embodiment of a feedback control loop in self-adaptive software systems, but the complete removal of humans from the control loop has not been thoroughly debated. One of the reasons is that, software systems are social-technical systems, and as such, humans need to be considered right from the inception of such systems, otherwise their deployment is bound to fail. However, as software self-adaptation progresses, enabling to place higher assurances on the deployment of these systems to the point humans become dispensable, some ethical questions need to be raised. Similar questions have been raised in past when the first automatic systems became intrinsic to the industrial fabric. The difference between then and now is that then the impact was confined to portions of the society, but now the implications are much wider, if we consider, in particular, software systems that are able to change themselves. If humans are not aware of those changes, and their implications, humans cease to be in tune with the system they are operating, and inevitably accidents will ensue. The point of no return in self-adaptive software systems refers to the moment in their technical maturity when any human involvement with the operation of a system is perceived to create more harm than benefit. Confronted with this situation, software engineers need start asking themselves some basic ethical questions. Do we really need to consider humans as an integral part of self-adaptive software systems? If humans are removed from the control loop, what kind of assurances will be needed for society to accept such systems?
- Jesper Andersson, Rogério de Lemos, Sam Malek, and Danny Weyns. 2009. Modeling Dimensions of Self-Adaptive Software Systems. In Software Engineering for Self-Adaptive Systems. Springer, 27--47.Google Scholar
- D. Dörner. 1997. The Logic of Failure: Recognizing and Avoiding Error in Complex Situations. Addison-Wesley Pub.Google Scholar
- Jack Nicas, Natalie Kitroeff, David Gelles, and James Glanz. 2019. Boeing Built Deadly Assumptions Into 737 Max, Blind to a Late Design Change. The New York Times (2019). https://www.nytimes.com/2019/06/01/business/boeing-737-max-crash.html [accessed: 23.01.2020].Google Scholar
- Ashley Nunes, Bryan Reimer, and Joseph Coughlin. 2018. People must retain control of autonomous vehicles. Nature 556 (04 2018), 169--171. Google ScholarCross Ref
- John Rushby. 2001. Modeling the Human in Human Factors. In Computer Safety, Reliability and Security, Udo Voges (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 86--91.Google Scholar
- Jeff Wise. 2011. What Really Happened Aboard Air France 447. Popular Mechanics (2011). https://www.popularmechanics.com/flight/a3115/what-really-happened-aboard-air-france-447-6611877/ [accessed: 23.01.2020].Google Scholar
Index Terms
- Human in the loop: what is the point of no return?
Recommendations
Extending MAPE-K to support human-machine teaming
SEAMS '22: Proceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing SystemsThe MAPE-K feedback loop has been established as the primary reference model for self-adaptive and autonomous systems in domains such as autonomous driving, robotics, and Cyber-Physical Systems. At the same time, the Human Machine Teaming (HMT) paradigm ...
Algorithms as partners in crime: A lesson in ethics by design
AbstractThe human in the loop is often advocated as a panacea against concerns about AI-powered machines, which increasingly take decisions of consequence in all realms of life. However, can we rely on humans to prevent unethical decisions by ...
Highlights- Our incentivized experiment examines ethical decision-making in human–machine teams.
Managing change in self-adaptive software systems
CompSysTech '11: Proceedings of the 12th International Conference on Computer Systems and TechnologiesSelf-adaptive software systems are expected to deal with change during their execution with limited or no human interaction. Therefore a better understanding of change and how it could be managed is a crucial step for achieving self-adaptation. Most of ...
Comments