Devleena Das
ABSTRACT
Intelligent decision support (IDS) systems leverage artificial intelligence techniques to generate recommendations that guide human users through the decision making phases of a task. However, a key challenge is that IDS systems are not perfect, and in complex real-world scenarios may produce suboptimal output or fail to work altogether. The field of explainable AI (XAI) has sought to develop techniques that improve the interpretability of black-box systems. While most XAI work has focused on single-classification tasks, the subfield of explainable AI planning (XAIP) has sought to develop techniques that make sequential decision making AI systems explainable to domain experts. Critically, prior work in applying XAIP techniques to IDS systems has assumed that the plan being proposed by the planner is always optimal, and therefore the action or plan being recommended as decision support to the user is always optimal. In this work, we examine novice user interactions with a non-robust IDS system – one that occasionally recommends suboptimal actions, and one that may become unavailable after users have become accustomed to its guidance. We introduce a new explanation type, subgoal-based explanations, for plan-based IDS systems, that supplements traditional IDS output with information about the subgoal toward which the recommended action would contribute. We demonstrate that subgoal-based explanations lead to improved user task performance in the presence of IDS recommendations, improve user ability to distinguish optimal and suboptimal IDS recommendations, and are preferred by users. Additionally, we demonstrate that subgoal-based explanations enable more robust user performance in the case of IDS failure, showing the significant benefit of training users for an underlying task with subgoal-based explanations.
- Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE access 6(2018), 52138–52160.Google Scholar
- Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. 2020. Debugging tests for model explanations. arXiv preprint arXiv:2011.05429(2020).Google Scholar
- Vicky Arnold, Philip A Collier, Stewart A Leech, and Steve G Sutton. 2004. Impact of intelligent decision aids on expert and novice decision-makers’ judgments. Accounting & Finance 44, 1 (2004), 1–26.Google ScholarCross Ref
- Gerard Canal, Senka Krivic, Paul Luff, and Andrew Coles. 2021. Task Plan verbalizations with causal justifications. In ICAPS 2021 Workshop on Explainable AI Planning (XAIP).Google Scholar
- Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. NeurIPS (2019).Google Scholar
- Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. 2020. The Emerging Landscape of Explainable Automated Planning & Decision Making.. In IJCAI. 4803–4811.Google Scholar
- Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. 2017. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. arXiv preprint arXiv:1701.08317(2017).Google Scholar
- Gergely Csibra and György Gergely. 2007. ‘Obsessed with goals’: Functions and mechanisms of teleological interpretation of actions in humans. Acta psychologica 124, 1 (2007), 60–78.Google Scholar
- Konrad Czechowski, Tomasz Odrzygóźdź, Marek Zbysiński, Michał Zawalski, Krzysztof Olejnik, Yuhuai Wu, Lukasz Kucinski, and Piotr Miłoś. 2021. Subgoal Search For Complex Reasoning Tasks. NeurIPS (2021).Google Scholar
- Devleena Das, Siddhartha Banerjee, and Sonia Chernova. 2021. Explainable ai for robot failures: Generating explanations that improve user assistance in fault recovery. In International Conference on Human-Robot Interaction. 351–360.Google ScholarDigital Library
- Devleena Das and Sonia Chernova. 2020. Leveraging rationales to improve human task performance. In IUI. 510–518.Google Scholar
- Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608(2017).Google Scholar
- Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. 2019. Automated rationale generation: a technique for explainable AI and its effects on human perceptions. In International Conference on Intelligent User Interfaces. 263–274.Google ScholarDigital Library
- Patrick Eyerich, Robert Mattmüller, and Gabriele Röger. 2009. Using the context-enhanced additive heuristic for temporal and numeric planning. In ICAPS.Google Scholar
- Jinyue Feng, Chantal Shaib, and Frank Rudzicz. 2020. Explainable clinical decision support from text. In EMNLP. 1478–1489.Google Scholar
- Maria Fox and Derek Long. 2003. PDDL2. 1: An extension to PDDL for expressing temporal planning domains. Journal of artificial intelligence research 20 (2003), 61–124.Google ScholarDigital Library
- Sachin Grover, Sailik Sengupta, Tathagata Chakraborti, Aditya Prasad Mishra, and Subbarao Kambhampati. 2020. RADAR: automated task planning for proactive decision support. Human–Computer Interaction 35, 5-6 (2020), 387–412.Google Scholar
- Stephanie Guerlain, Donald E Brown, and Christina Mastrangelo. 2000. Intelligent decision support systems. In SMC, Vol. 3. IEEE.Google ScholarCross Ref
- Francisco Gutiérrez, Xavier Ochoa, Karsten Seipp, Tom Broos, and Katrien Verbert. 2019. Benefits and Trade-Offs of Different Model Representations in Decision Support Systems for Non-expert Users. In INTERACT, David Lamas, Fernando Loizides, Lennart Nacke, Helen Petrie, Marco Winckler, and Panayiotis Zaphiris (Eds.). Springer International Publishing, 576–597.Google Scholar
- Jörg Hoffmann and Daniele Magazzeni. 2019. Explainable AI planning (XAIP): overview and the case of contrastive explanation. Reasoning Web: Explainable Artificial Intelligence (2019), 277–282.Google Scholar
- Matteo Iovino, Edvards Scukins, Jonathan Styrud, Petter Ögren, and Christian Smith. 2020. A survey of behavior trees in robotics and ai. arXiv preprint arXiv:2005.05842(2020).Google Scholar
- Richard W Jones, James E Mateer, and Michael J Harrison. 2019. Malfunction transparency in clinical decision support systems: a classification approach. In ICIEA. IEEE, 1354–1359.Google Scholar
- Leslie Pack Kaelbling and Tomás Lozano-Pérez. 2010. Hierarchical planning in the now. In Workshops at AAAI.Google Scholar
- Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML. PMLR, 2668–2677.Google Scholar
- Chinho Lin, Chun Mei Lin, Binshan Lin, and Ming-Chin Yang. 2009. A decision support system for improving doctors’ prescribing behavior. Expert Systems with Applications 36, 4 (2009), 7975–7984.Google ScholarDigital Library
- Yuxin Liu, Qiguang Chen, Ke Jin, and Zhanhao Zhang. 2020. Planning for Overcooked Game with PDDL. International Core Journal of Engineering 6, 12 (2020), 315–325.Google Scholar
- Jessica P Machado, Xuan T Lam, and Jung-Wei Chen. 2018. Use of a clinical decision support tool for the management of traumatic dental injuries in the primary dentition by novice and expert clinicians. Dental Traumatology 34, 2 (2018), 120–128.Google ScholarCross Ref
- Allen Newell, Herbert Alexander Simon, 1972. Human problem solving. Vol. 104. Prentice-hall Englewood Cliffs, NJ.Google Scholar
- P Russel Norvig and S Artificial Intelligence. 2002. A modern approach. Prentice Hall Upper Saddle River, NJ, USA: Rani, M., Nayak, R., & Vyas, OP (2015). An ontology-based adaptive personalized e-learning system, assisted by software agents on cloud storage. Knowledge-Based Systems 90 (2002), 33–48.Google Scholar
- Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 112–121.Google ScholarCross Ref
- KN Papamichail and S French. 2000. Decision support in nuclear emergencies. J. of hazardous materials 71, 1-3 (2000), 321–342.Google Scholar
- Shailendra Rathore, Vincenzo Loia, and Jong Hyuk Park. 2018. SpamSpotter: An efficient spammer detection framework based on intelligent decision support system on Facebook. Applied Soft Computing 67 (2018), 920–932.Google ScholarDigital Library
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should I trust you?" Explaining the predictions of any classifier. In Proc. of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135–1144.Google ScholarDigital Library
- Silvia Richter, Malte Helmert, and Matthias Westphal. 2008. Landmarks Revisited.. In AAAI, Vol. 8. 975–982.Google Scholar
- Germán González Rodríguez, Jose M Gonzalez-Cava, and Juan Albino Méndez Pérez. 2020. An intelligent decision support system for production planning based on machine learning. Journal of Intelligent Manufacturing 31, 5 (2020), 1257–1273.Google ScholarDigital Library
- Roger C Schank and Robert P Abelson. 2013. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press.Google Scholar
- Bastian Seegebarth, Felix Müller, Bernd Schattenberg, and Susanne Biundo. 2012. Making hybrid plans more clear to human users-a formal approach for generating sound explanations. In Twenty-second international conference on automated planning and scheduling.Google ScholarDigital Library
- Sarath Sreedharan, Siddharth Srivastava, David Smith, and Subbarao Kambhampati. 2019. Why Can’t You Do That HAL? Explaining Unsolvability of Planning Tasks. In IJCAI.Google Scholar
- Reed T Sutton, David Pincock, Daniel C Baumgart, Daniel C Sadowski, Richard N Fedorak, and Karen I Kroeker. 2020. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ digital medicine 3, 1 (2020), 1–10.Google Scholar
- Aaquib Tabrez, Shivendra Agrawal, and Bradley Hayes. 2019. Explanation-based reward coaching to improve human performance via reinforcement learning. In HRI. IEEE, 249–257.Google Scholar
- Robin R Vallacher and Daniel M Wegner. 1987. What do people think they’re doing? Action identification and human behavior.Psychological review 94, 1 (1987), 3.Google Scholar
- Karthik Valmeekam, Sarath Sreedharan, Sailik Sengupta, and Subbarao Kambhampati. 2020. RADAR-X: An Interactive Interface Pairing Contrastive Explanations with Revised Plan Suggestions. arXiv preprint arXiv:2011.09644(2020).Google Scholar
- Seán Walsh, Evelyn EC de Jong, Janna E van Timmeren, Abdalla Ibrahim, Inge Compter, Jurgen Peerlings, Sebastian Sanduleanu, Turkey Refaee, Simon Keek, Ruben THM Larue, 2019. Decision support systems in oncology. JCO clinical cancer informatics 3 (2019), 1–9.Google Scholar
- Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. 2021. Too Many Cooks: Bayesian Inference for Coordinating Multi-Agent Collaboration. Topics in Cognitive Science 13, 2 (2021), 414–432.Google ScholarCross Ref
- Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. 2019. Interpreting cnns via decision trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6261–6270.Google ScholarCross Ref
- Yu Zhang, Sarath Sreedharan, Anagha Kulkarni, Tathagata Chakraborti, Hankz Hankui Zhuo, and Subbarao Kambhampati. 2017. Plan explicability and predictability for robot task planning. In 2017 IEEE international conference on robotics and automation (ICRA). IEEE, 1313–1320.Google ScholarDigital Library
- Zoe Y Zhuang, Leonid Churilov, Frada Burstein, and Ken Sikaris. 2009. Combining data mining and case-based reasoning for intelligent decision support for pathology ordering by general practitioners. European Journal of Operational Research 195, 3 (2009), 662–675.Google ScholarCross Ref
Index Terms
- Subgoal-Based Explanations for Unreliable Intelligent Decision Support Systems
Recommendations
Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing SystemsThe field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to ...
Human-Centered Evaluation of Explanations in AI-Assisted Decision-Making
IUI '24 Companion: Companion Proceedings of the 29th International Conference on Intelligent User InterfacesAI explanations have been increasingly used to help people better utilize AI recommendations in AI-assisted decision making. While numerous technical transparency approaches have been established, a human-centered perspective is needed for understanding ...
Explanation Ontology: A Model of Explanations for User-Centered AI
The Semantic Web – ISWC 2020AbstractExplainability has been a goal for Artificial Intelligence (AI) systems since their conception, with the need for explainability growing as more complex AI models are increasingly used in critical, high-stakes settings such as healthcare. ...
Comments