Ronal Singh
Liz Sonenberg
Eduardo Velloso
Frank Vetere
Paul Dourish
Skip Abstract Section
Skip Supplemental Material SectionAbstract
In this article, we show that explanations of decisions made by machine learning systems can be improved by not only explaining why a decision was made but also explaining how an individual could obtain their desired outcome. We formally define the concept of directive explanations (those that offer specific actions an individual could take to achieve their desired outcome), introduce two forms of directive explanations (directive-specific and directive-generic), and describe how these can be generated computationally. We investigate people’s preference for and perception toward directive explanations through two online studies, one quantitative and the other qualitative, each covering two domains (the credit scoring domain and the employee satisfaction domain). We find a significant preference for both forms of directive explanations compared to non-directive counterfactual explanations. However, we also find that preferences are affected by many aspects, including individual preferences and social factors. We conclude that deciding what type of explanation to provide requires information about the recipients and other contextual information. This reinforces the need for a human-centered and context-specific approach to explainable AI.
Supplemental Material
Available for Download
pdf
3579363.app.pdf (253.4 KB)
Supplementary material
- [1] . 2018. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access 6 (2018), 52138–52160.Google Scholar
Cross Ref
- [2] . 2010. The inverse classification problem. Journal of Computer Science and Technology 25, 3 (2010), 458–468.Google ScholarDigital Library
- [3] . 2018. Interpretable machine learning in healthcare. In Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (BCB’18). Association for Computing Machinery, New York, NY, 559–560.Google ScholarDigital Library
- [4] . 2020. Explanation in AI and law: Past, present and future. Artif. Intell. 289 (
Dec. 2020), 103387.Google ScholarCross Ref - [5] . 2017. Leveraging human routine models to detect and generate human behaviors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI’17). Association for Computing Machinery, New York, NY, 6683–6694.Google ScholarDigital Library
- [6] . 2020. The hidden assumptions behind counterfactual explanations and principal reasons. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT*’20). Association for Computing Machinery, New York, NY, 80–89.Google ScholarDigital Library
- [7] . 1957. A Markovian decision process. Journal of Mathematics and Mechanics 6, 5 (1957), 679–684.Google Scholar
- [8] . 2018. “It’s reducing a human being to a percentage”: Perceptions of justice in algorithmic decisions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). Association for Computing Machinery, New York, NY, 1–14.Google ScholarDigital Library
- [9] . 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI), Vol. 8. cs.columbia.edu, 8–13.Google Scholar
- [10] . 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101.Google ScholarCross Ref
- [11] . 2012. A survey of Monte Carlo tree search methods. IEEE Trans. Comput. Intell. AI Games 4, 1 (
March 2012), 1–43.Google ScholarCross Ref - [12] . 2014. Landmark-based plan distance measures for diverse planning. ICAPS 24 (
May 2014), 56–64.Google ScholarCross Ref - [13] . 2022. Machine learning interpretability for a stress scenario generation in credit scoring based on counterfactuals. Expert Syst. Appl. 202 (
Sept. 2022), 117271.Google ScholarDigital Library - [14] . 2018. Woulda, coulda, shoulda: Counterfactually-guided policy search. (
Nov. 2018).arxiv:1811.06272 [cs.LG]Google Scholar - [15] . 2011. Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6, 1 (
Jan. 2011), 3–5.Google ScholarCross Ref - [16] . 2020. Explainable AI in fintech risk management. Front. Artif. Intell. 3 (
April 2020), 26.Google ScholarCross Ref - [17] . 2016. Counterfactual thought. Annual Review of Psychology 67 (2016), 135–157.Google ScholarCross Ref
- [18] . 2019. Counterfactuals in explainable artificial intelligence (XAI): Evidence from human reasoning. Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI’19 Macao, 10-16 August 2019), ijcai.org, 6276–6282.Google Scholar
- [19] . 2017. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. (
Jan. 2017).arxiv:1701.08317 [cs.AI]Google Scholar - [20] . 2020. Explainability as a non-functional requirement: Challenges and recommendations. Requirements Engineering 25, 4 (
Dec. 2020), 493–514.Google ScholarDigital Library - [21] . 2014. Thematic analysis. (2014), 1947–1952. Google ScholarCross Ref
- [22] . 2020. Causal modeling for fairness in dynamical systems. In Proceedings of the 37th International Conference on Machine Learning(
Proceedings of Machine Learning Research , Vol. 119), and (Eds.). PMLR, 2185–2195.Google Scholar - [23] . 2020. Multi-objective counterfactual explanations. In Parallel Problem Solving from Nature (PPSN XVI). Springer International Publishing, 448–469.Google Scholar
- [24] . 2018. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation (EC’18 Ithaca, NY, USA, June 18-22, 2018), Éva Tardos, Edith Elkind, and Rakesh Vohra (Eds.). ACM, 55–70. Google ScholarDigital Library
- [25] . 2021. Prototype-based counterfactual explanation for causal classification. (
May 2021).arxiv:2105.00703 [cs.LG]Google Scholar - [26] . 2017. Slave to the algorithm: Why a right to an explanation is probably not the remedy you are looking for. Duke L. & Tech. Rev. 16 (2017), 18.Google Scholar
- [27] . 2019. Automated rationale generation: A technique for explainable AI and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI’19). Association for Computing Machinery, New York, NY, 263–274.Google ScholarDigital Library
- [28] . 2018. Analysis of classifiers’ robustness to adversarial perturbations. Machine Learning 107, 3 (2018), 481–508.Google ScholarDigital Library
- [29] . 2017. Explainable planning. (
Sept. 2017).arxiv:1709.10256 [cs.AI]Google Scholar - [30] . 2013. A concise introduction to models and methods for automated planning. Synthesis Lectures on Artificial Intelligence and Machine Learning 8, 1 (2013), 1–141.Google ScholarCross Ref
- [31] . 2019. A survey of methods for explaining black box models. ACM Computing Surveys (CSUR) 51, 5 (2019), 93.Google ScholarDigital Library
- [32] . 2019. A systematic method to understand requirements for explainable AI (XAI) systems. In Proceedings of the IJCAI Workshop on eXplainable Artificial Intelligence (XAI’19), Vol. 11. dais-ita.org.Google Scholar
- [33] . 2020. Causes and explanations: A structural-model approach. Part I: Causes. Br. J. Philos. Sci. (2020).Google Scholar
- [34] . 2016. Strategic classification. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science (ITCS’16). Association for Computing Machinery, New York, NY, 111–122.Google ScholarDigital Library
- [35] . 2017. Explaining explanation, part 1: Theoretical foundations. IEEE Intelligent Systems 32, 3 (2017), 68–73.Google ScholarDigital Library
- [36] . 2017. What do we need to build explainable AI systems for the medical domain? (
Dec. 2017).arxiv:1712.09923 [cs.AI]Google Scholar - [37] . 2007. Causality and counterfactuals in the situation calculus. J. Logic Comput. 17, 5 (
Oct. 2007), 939–953.Google ScholarDigital Library - [38] . 2021. Algorithmic recourse: From counterfactual explanations to interventions. (2021), 353–362. Google ScholarDigital Library
- [39] . 2020. Model-agnostic counterfactual explanations for consequential decisions. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics(
Proceedings of Machine Learning Research , Vol. 108), and (Eds.). PMLR, 895–905.Google Scholar - [40] . 2020. Algorithmic recourse under imperfect causal knowledge: A probabilistic approach. (
June 2020).arxiv:2006.06831 [cs.LG]Google Scholar - [41] . 2020. Reshaping diverse planning. AAAI 34, 06 (
April 2020), 9892–9899.Google ScholarCross Ref - [42] . 2018. Human decisions and machine predictions. Q. J. Econ. 133, 1 (
Feb. 2018), 237–293.Google Scholar - [43] . 2021. A causal perspective on meaningful and robust algorithmic recourse. (
July 2021).arxiv:2107.07853 [stat.ML]Google Scholar - [44] . 2019. Model-based contrastive explanations for explainable planning. In ICAPS 2019 Workshop on Explainable AI Planning (XAIP’19). AAAI Press, 9.Google Scholar
- [45] . 2013. Counterfactuals. John Wiley & Sons.Google Scholar
- [46] . 2020. Questioning the AI: Informing design practices for explainable AI user experiences. arXiv preprint arXiv:2001.02478 (2020).Google Scholar
- [47] . 2009. Assessing demand for intelligibility in context-aware applications. In Ubiquitous Computing, 11th International Conference (UbiComp’09), Proceedings (ACM International Conference Proceeding Series), , , and (Eds.). ACM, 195–204. Google ScholarDigital Library
- [48] . 2018. The mythos of model interpretability. Commun. ACM 61, 10 (2018), 36–43. Google ScholarDigital Library
- [49] . 2019. A grounded interaction protocol for explainable artificial intelligence. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS’19). International Foundation for Autonomous Agents and Multiagent Systems, 1033–1041.Google ScholarDigital Library
- [50] . 2020. Explainable reinforcement learning through a causal lens. (2020), 2493–2500. https://aaai.org/ojs/index.php/AAAI/article/view/5631.Google Scholar
- [51] . 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (2019), 1–38.Google ScholarCross Ref
- [52] . 2021. Contrastive explanation: A structural-model approach. Knowl. Eng. Rev. 36 (2021), e14.Google ScholarCross Ref
- [53] . 2020. Interpretable Machine Learning. Lulu.com.Google Scholar
- [54] . 2020. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 607–617.Google ScholarDigital Library
- [55] . 2021. Consequence-aware sequential counterfactual generation. (
April 2021).arxiv:2104.05592 [cs.LG]Google Scholar - [56] . 2017. Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods 16, 1 (2017), 1609406917733847.Google ScholarCross Ref
- [57] . 2019. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning (ICML’19). proceedings.mlr.press, 4881–4890.Google Scholar
- [58] . 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(
CHI’21 , Article 237). Association for Computing Machinery, New York, NY, 1–52.Google ScholarDigital Library - [59] . 2020. FACE: Feasible and actionable counterfactual explanations. In AAAI/ACM Conference on AI, Ethics, and Society (AIES’20, New York, NY, USA, February 7-8, 2020), Annette N. Markham, Julia Powles, Toby Walsh, and Anne L. Washington (Eds.). ACM, 344–350. Google ScholarDigital Library
- [60] . 2014. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.Google Scholar
- [61] . 2018. Explanations as mechanisms for supporting algorithmic transparency. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). Association for Computing Machinery, New York, NY, 1–13.Google ScholarDigital Library
- [62] . 2016. “Why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, , , , , , and (Eds.). ACM, 1135–1144. Google ScholarDigital Library
- [63] . 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI’18), the 30th innovative Applications of Artificial Intelligence (IAAI’18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18, New Orleans, Louisiana, USA, February 2-7, 2018), and (Eds.). AAAI Press, 1527–1535. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16982.Google ScholarCross Ref
- [64] . 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206–215.Google ScholarCross Ref
- [65] . 2019. Efficient search for diverse coherent explanations. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT’19, Atlanta, GA, USA, January 29-31, 2019), and (Eds.). ACM, 20–28. Google ScholarDigital Library
- [66] . 2018. The intuitive appeal of explainable machines. Fordham L. Rev. 87 (2018), 1085.Google Scholar
- [67] . 2019. CERTIFAI: Counterfactual explanations for robustness, transparency, interpretability, and fairness of artificial intelligence models. (
May 2019).arxiv:1905.07857 [cs.LG]Google Scholar - [68] . 2020. One explanation does not fit all: The promise of interactive explanations for machine learning transparency. CoRR abs/2001.09734 (2020).
arxiv:2001.09734 https://arxiv.org/abs/2001.09734.Google Scholar - [69] . 2022. Explainable human–AI interaction: A planning perspective. Synthesis Lectures on Artificial Intelligence and Machine Learning 16, 1 (
Jan. 2022), 1–184.Google ScholarCross Ref - [70] . 2007. Domain independent approaches for finding diverse plans. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI’07, Hyderabad, India, January 6-12, 2007), (Ed.). 2016–2022. http://ijcai.org/Proceedings/07/Papers/325.pdf.Google Scholar
- [71] . 2018. Reinforcement Learning: An Introduction (2nd ed.). MIT Press.Google ScholarDigital Library
- [72] . 1980. Meeting the Equal Credit Opportunity Act’s specificity requirement: Judgmental and statistical scoring systems. Buff. L. Rev. 29 (1980), 73.Google Scholar
- [73] . 2018. Interpretable to whom? A role-based model for analyzing interpretable machine learning systems. CoRR abs/1806.07552 (2018).
arxiv:1806.07552 http://arxiv.org/abs/1806.07552.Google Scholar - [74] . 2021. Counterfactual explanations in sequential decision making under uncertainty. (
July 2021).arxiv:2107.02776 [cs.LG]Google Scholar - [75] . 2020. Decisions, counterfactual explanations and strategic behavior. (
Feb. 2020).arxiv:2002.04333 [cs.LG]Google Scholar - [76] . 2019. Actionable recourse in linear classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT’19, Atlanta, GA, USA, January 29-31, 2019), and (Eds.). ACM, 10–19. Google ScholarDigital Library
- [77] . 2020. The philosophical basis of algorithmic recourse. In Conference on Fairness, Accountability, and Transparency (FAccT’20, Barcelona, Spain, January 27-30, 2020), , , , , , and (Eds.). ACM, 284–293. Google ScholarDigital Library
- [78] . 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. J. L. & Tech. 31 (2017), 841.Google Scholar
- [79] . 2019. Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19, New York, NY, USA, May 2019). Glasgow, 1–15.Google ScholarDigital Library
- [80] . 2010. When in Rome: The role of culture & context in adherence to robot recommendations. In 2010 5th ACM/IEEE International Conference on Human-robot Interaction (HRI’10). ieeexplore.ieee.org, 359–366.Google ScholarCross Ref
Index Terms
- Directive Explanations for Actionable Explainability in Machine Learning Applications
Recommendations
Explainability Metrics and Properties for Counterfactual Explanation Methods
Explainable and Transparent AI and Multi-Agent SystemsAbstractThe increasing application of Explainable AI (XAI) methods to enhance the transparency and trustworthiness of AI systems designates the need to quantitatively assess and analyze the theoretical and behavioral characteristics of explanations ...
The Privacy Issue of Counterfactual Explanations: Explanation Linkage Attacks
Black-box machine learning models are used in an increasing number of high-stakes domains, and this creates a growing need for Explainable AI (XAI). However, the use of XAI in machine learning introduces privacy risks, which currently remain largely ...
Counterfactual Explanations for Reinforcement Learning Agents
AAMAS '23: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent SystemsReinforcement learning (RL) algorithms often use neural networks to represent agent's policy, making them difficult to interpret. Counterfactual explanations are human-friendly explanations which offer users actionable advice on how to change their ...
Comments