Matt Bishop
Carrie Gates
ABSTRACT
The information security community is haunted by the failure of an appropriate break-the-glass access control at the United States Center for Disease Control that led to an estimated additional 1.2 million deaths in North America in 2036. In this paper we review what caused the security failures in this system and argue that, by combining human intelligence with multiple technological approaches to create a system that emphasizes human approaches to guide analysis, the failures that occurred will not recur. We also leverage people and technologies to identify and fill gaps in the training data to minimize the threat of unexpected events. While we use this scenario as our running example, we note that our approach is generalizable to a broader problem space where machine learning approaches have been deployed to make decisions.
- Saul D. Alinsky. 1989. Rules for Radicals. Vintage Books, New York, NY, USA.Google Scholar
- George S. Avrunin, Lori A. Clarke, Leon J. Osterweil, Stefan C. Christov, Bin Chen, Elizabeth A. Henneman, Philip L. Henneman, Lucinda Cassells, and Wilson Mertens. 2010. Experience Modeling and Analyzing Medical Processes: UMass/Baystate Medical Safety Project Overview. In Proceedings of the First ACM International Health Informatics Symposium. ACM, New York, NY, USA, 316--325. Google ScholarDigital Library
- Matt Bishop, Carrie Gates, and Karl Levitt. 2018. Augmenting Machine Learning with Argumentation. In Proceedings of the 2018 New Security Paradigms Workshop. ACM, New York, NY, USA, 50--55. Google ScholarDigital Library
- Kenneth T. Broda-Bahm. 1995. Counterfactual Possibilities: Constructing Counter-to-Fact Causal Claims. Contemporary Argumentation and Debate 16 (1995), 73--85.Google Scholar
- Stefan Christov, Bin Chen, George S. Avrunin, Lori A. Clarke, Leon J. Osterweil, David Brown, Lucinda Cassells, and Wilson Mertens. 2007. Rigorously Defining and Analyzing Medical Processes: An Experience Report. In Proceedings of the 2007 International Conference on Model Driven Engineering Languages and Systems (Lecture Notes in Computer Science), Vol. 5002. Springer, Berlin, 118--131. Google ScholarDigital Library
- Phan Minh Dung. 1995. On the Acceptability of Arguments and Its Fundamental Role in Nonmonotonic Reasoning, Logic Programming and n-Person Games. Artificial Intelligence 77, 2 (Sept. 1995), 231--357. Google ScholarDigital Library
- Anna Ferreira, Ricardo Cruz-Correia, Luis Antunes, Pedro Farinha, E. Oliveira-Palhares, David W Chadwick, and Altamiro Costa-Pereira. 2006. How to Break Access Control in a Controlled Manner. In Proceedings of the 19th IEEE International Symposium on Computer-Based Medical Systems. IEEE, Los Alamitos, CA, USA, 847--854. Google ScholarDigital Library
- Amy Fowler and Will Kane. 2031. Dark Horse Healthcare Promotes Deep Learning Across Healthcare Industry. The New York Times (Sep. 30, 2031), B1, B4. {future publication}Google Scholar
- Emmanouil Georgakakis, Stefanos A Nikolidakis, Dimitrios D Vergados, and Christos Douligeris. 2011. Spatio Temporal Emergency Role Based Access Control (STEM-RBAC): A Time and Location Aware Role Based Access Control Model with a Break the Glass Mechanism. In Proceedings of the 2011 IEEE Symposium on Computers and Communications. IEEE, Los Alamitos, CA, USA, 764--770. Google ScholarDigital Library
- Sergio Alejandro Gómez and Carlos Iván Chesñevar. 2004. Integrating Defeasible Argumentation and Machine Learning Techniques. Technical Report arXiv:cs/0402057v2 {cs.AI}. CoRR.Google Scholar
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, Vol. 27. Curran Associates, Inc., Red Hook, NY, USA, 2672--2680. Google ScholarDigital Library
- Thomas Gradgrind. 2024. Supplementing Limited Training Data with Human Intuition in Deep Learning. In Proceedings of the 20th International Conference on Machine Learning and Data Mining. IEEE Computer Society, Los Alamitos, CA, USA, 129--147. {future publication}Google Scholar
- C. Cordell Green. 1969. Theorem Proving by Resolution as a Basis for Question-Answering Systems. In Proceedings of the Fourth Annual Machine Intelligence Workshop. Edinburgh University Press, Edinburgh, Scotland, 183--208.Google Scholar
- Cormac Herley and Wolter Pieters. 2015. "If You Were Attacked, You'd Be Sorry": Counterfactuals as Security Arguments. In Proceedings of the 2015 New Security Paradigms Workshop. ACM, New York, NY, USA, 112--123. Google ScholarDigital Library
- Mycroft Holmes and Manuel Garcia O'Kelly-Davis. 2025. Logical Operations (Lo-gOps) -- A New Approach to Machine Learning. Advances in Neural Information Processing Systems 16, 5 (2025), 1271--1283. {future publication}Google Scholar
- Odis Johnson and Michael Wagner. 2017. Equalizers or Enablers of Inequality? A Counterfactual Analysis of Racial and Residential Test Score Gaps in Year-Round and Nine-Month Schools. Annals of the American Academy of Political and Social Science 674, 1 (2017), 240--261.Google ScholarCross Ref
- Wyoming Knott and Adam Selene. 2026. Leveraging Lessons from Expert Systems to Improve Deep Learning. Expert Systems with Applications 78, 12 (2026), 29--39. {future publication}Google Scholar
- Ruth Leavitt, Jeremy Stone, and Mark Hall. 2028. Deep Learning for Health Record Access Control. IEEE Journal of Biomedical and Health Informatics 32, 1 (2028), 4--21. {future publication}Google Scholar
- Ming Li, Shucheng Yu, Kui Ren, and Wenjing Lou. 2010. Securing Personal Health Records in Cloud Computing: Patient-Centric and Fine-Grained Data Access Control in Multi-Owner Settings. In Proceedings of the 2010 International Conference on Security and Privacy in Communication Systems (Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering), Vol. 50. Springer, Berlin, Heidelberg, Germany, 89--106.Google Scholar
- Srdjan Marinovic, Naranker Dulay, and Morris Sloman. 2014. Rumpole: An Introspective Break-Glass Access Control Language. ACM Transactions on Information and System Security 17, 1 (2014), 2. Google ScholarDigital Library
- Andrew Martin. 2028. Testing AI-Based Access Control for Electronic Health Records. IEEE Proceedings on Software Development 165, 1 (2028), 16--23. {future publication}Google Scholar
- Leonard McCoy and Beverly Crusher. 2037. Failures in 2036 Infection Response. NIH Publication 37-0071. National Institute of Health, Washington, DC. {future publication}Google Scholar
- Martin Možina, Jure Žabkar, and Ivan Bratko. 2007. Argument Based Machine Learning. Artificial Intelligence 171, 10--15 (July 2007), 922--937. Google ScholarDigital Library
- Leland Gerson Neuberg. 2003. Review of 'Causality: Models, Reasoning, and Inference'. Econometric Theory 19, 4 (Aug. 2003), 675--685.Google Scholar
- Robert M. O'Keefe and Daniel E. O'Leary. 1993. Expert System Verification and Validation: A Survey and Tutorial. Artificial Intelligence Review 7, 1 (Feb. 1993), 3--42.Google ScholarCross Ref
- R. Daneel Olivaw. 2034. A Survey of Coordinated Attacks and Collaborative Intrusion Detection. Computers & Security 53, 2 (2034), 117--138. {future publication}Google Scholar
- Leon J. Osterweil, Matt Bishop, Heather M. Conboy, Huong Phan, Borislava I. Simidchieva, George S. Avrunin, Lori A. Clarke, and Sean Peisert. 2017. Iterative Analysis to Improve Key Properties of Critical Human-Intensive Processes: An Election Security Example. ACM Transactions on Privacy and Security 20, 2 (March 2017), 5:1--5:31. Google ScholarDigital Library
- Anna Palczewska, Jan Palczewski, Richard Mrachese Robinson, and Daniel Neagu. 2014. Integration of Reusable Systems. Advances in Intelligent Systems and Computing, Vol. 263. Springer, Cham, Switzerland, Chapter Interpreting Random Forest Classification Models Using a Feature Contribution Method, 193--218.Google Scholar
- Judea Pearl. 2009. Causality: Models, Reasoning and Inference (2nd ed.). Cambridge University Press, Cambridge, UK. Google ScholarDigital Library
- Wolter Pieters. 2011. Explanation and Trust: What to Tell the User in Security and AI? Ethics and Information Technology 13, 1 (March 2011), 53--64. Google ScholarDigital Library
- Dean Povey. 1999. Optimistic Security: A New Access Control Paradigm. In Proceedings of the 1999 New Security Paradigms Workshop. ACM, New York, NY, USA, 40--45. Google ScholarDigital Library
- Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. NN Features Off-the-Shelf: An Astounding Baseline for Recognition. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE Computer Society, Los Alamitos, CA, USA, 512--519. Google ScholarDigital Library
- Jay Score. 2024. Applying SVMs to Predict Appropriate Access Control Policies. IEEE Transactions on Dependable and Secure Computing 21, 5 (2024), 533--545. {future publication}Google Scholar
- Hari Seldon and Gaal Dornick. 2023. Generative Adversarial Networks in Society: A Survey. Proceedings of the 17th International AAAI Conference on Web and Social Media 23 (2023), 505--514. {future publication}Google Scholar
- Rakshith Shetty, Bernt Schiele, and Mario Fritz. 2018. A4NT: Author Attribute Anonymity by Adversarial Training of Neural Machine Translation. In Proceedings of the 27th USENIX Security Symposium. USENIX Association, Berkeley, CA, USA, 1633--1650. Google ScholarDigital Library
- Hermann Strangelove. 2023. Naive-Bayes to Break-the-Glass in RBAC-Administered Databases. In Proceedings of the 39th Annual Computer Security Applications Conference. IEEE Computer Society, Los Alamitos, CA, USA, 1--10. {future publication}Google Scholar
- William Sutton. 2022. A Review of Break-the-Glass Scenarios and Real-World Consequences. Journal of American Medical Informatics Association 29, 4 (2022), 2--17. {future publication}Google Scholar
- Stephen E. Toulmin. 2003. The Uses of Argument. Cambridge University Press, Cambridge, UK.Google Scholar
- Sun Tzu. 1983. The Art of War. Delacorte Press, New York, NY, USA.Google Scholar
- David Wagner and Paolo Soto. 2002. Mimicry Attacks on Host-based Intrusion Detection Systems. In Proceedings of the Ninth ACM Conference on Computer and Communications Security. ACM, New York, NY, USA, 255--264. Google ScholarDigital Library
- Alan T. Wilson. 2015. Counterfactual Consent and the Use of Deception in Research. Bioethics 29, 7 (Sept. 2015), 470--477.Google ScholarCross Ref
- Leon Windscheid, Lynn Bowes-Sperry, Jens Mazei, and Michèle Morner. 2017. The Paradox of Diversity Initiatives: When Organizational Needs Differ from Employee Preferences. Journal of Business Ethics 145, 1 (Sept. 2017), 33--48.Google ScholarCross Ref
- Alexander Wise, Aaron G. Cass, Barbara Staudt Lerner, Eric K. McCall, Leon J. Osterweil, and Jr. Stanley M. Sutton. 2000. Using Little-JIL to Coordinate Agents in Software Engineering. In Proceedings of the Fifteenth IEEE International Conference on Automated Software Engineering. IEEE, Piscataway, NJ, USA, 155--163. Google ScholarDigital Library
- Quan-shi Zhang and Song-chun Zhu. 2018. Visual Interpretability for Deep Learning: A Survey. Frontiers of Information Technology & Electronic Engineering 19, 1 (Jan. 2018), 27--39.Google Scholar
- Didar Zowghi and Chad Coulin. 2005. Requirements Elicitation: A Survey of Techniques, Approaches, and Tools. In Engineering and Managing Software Requirements. Springer, Berlin, Heidelberg, 19--46.Google Scholar
Index Terms
- Augmenting Machine Learning with Argumentation
Recommendations
Computing Arguments and Attacks in Assumption-Based Argumentation
CaSAPI (Credulous and Skeptical Argumentation: Prolog Implementation) 3.0 determines the acceptability of claims, using the general-purpose framework of assumption-based argumentation, under the semantics of admissible extensions. This framework reduces ...
Inferring from Inconsistency in Preference-Based Argumentation Frameworks
Argumentation is a promising approach to handle inconsistent knowledge bases, based on the justification of plausible conclusions by arguments. Because of inconsistency, however, arguments may be defeated by counterarguments (or defeaters). The problem ...
A structured argumentation system with backing and undercutting
This work introduces Extended Defeasible Logic Programming (E-DeLP), a structured argumentation system enabling the expression of reasons for and against using defeasible rules. E-DeLPextends the formalism of Defeasible Logic Programming (DeLP) by ...
Comments