Dell Zhang
ABSTRACT
This full-day tutorial introduces modern techniques for practical uncertainty quantification specifically in the context of multi-class and multi-label text classification. First, we explain the usefulness of estimating aleatoric uncertainty and epistemic uncertainty for text classification models. Then, we describe several state-of-the-art approaches to uncertainty quantification and analyze their scalability to big text data: Virtual Ensemble in GBDT, Bayesian Deep Learning (including Deep Ensemble, Monte-Carlo Dropout, Bayes by Backprop, and their generalization Epistemic Neural Networks), Evidential Deep Learning (including Prior Networks and Posterior Networks), as well as Distance Awareness (including Spectral-normalized Neural Gaussian Process and Deep Deterministic Uncertainty). Next, we talk about the latest advances in uncertainty quantification for pre-trained language models (including asking language models to express their uncertainty, interpreting uncertainties of text classifiers built on large-scale language models, uncertainty estimation in text generation, calibration of language models, and calibration for in-context learning). After that, we discuss typical application scenarios of uncertainty quantification in text classification (including in-domain calibration, cross-domain robustness, and novel class detection). Finally, we list popular performance metrics for the evaluation of uncertainty quantification effectiveness in text classification. Practical hands-on examples/exercises are provided to the attendees for them to experiment with different uncertainty quantification methods on a few real-world text classification datasets such as CLINC150.
- Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. 2021. A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges. Information Fusion, Vol. 76 (Dec. 2021), 243--297. https://doi.org/10.1016/j.inffus.2021.05.008Google Scholar
Digital Library
- Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight Uncertainty in Neural Network. In Proceedings of the 32nd International Conference on Machine Learning. PMLR, 1613--1622.Google Scholar
- Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. 2020. Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 1356--1367.Google Scholar
- Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning. PMLR, 1050--1059.Google Scholar
- Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang Zhu. 2022. A Survey of Uncertainty in Deep Neural Networks. https://doi.org/10.48550/arXiv.2107.03342 arxiv: 2107.03342 [cs, stat]Google ScholarCross Ref
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML '17). JMLR.org, Sydney, NSW, Australia, 1321--1330.Google Scholar
- Zhen Guo, Zelin Wan, Qisheng Zhang, Xujiang Zhao, Feng Chen, Jin-Hee Cho, Qi Zhang, Lance M. Kaplan, Dong H. Jeong, and Audun Jøsang. 2022. A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learning. https://doi.org/10.48550/arXiv.2206.05675 arxiv: 2206.05675 [cs, math]Google ScholarCross Ref
- Reihaneh H. Hariri, Erik M. Fredericks, and Kate M. Bowers. 2019. Uncertainty in Big Data Analytics: Survey, Opportunities, and Challenges. Journal of Big Data, Vol. 6, 1 (June 2019), 44. https://doi.org/10.1186/s40537-019-0206-3Google ScholarCross Ref
- Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. 2022. Scaling Out-of-Distribution Detection for Real-World Settings. In Proceedings of the 39th International Conference on Machine Learning. PMLR, 8759--8773.Google Scholar
- Dan Hendrycks and Kevin Gimpel. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In International Conference on Learning Representations.Google Scholar
- Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. 2019. Deep Anomaly Detection with Outlier Exposure. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.Google Scholar
- Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian Active Learning for Classification and Preference Learning. https://doi.org/10.48550/arXiv.1112.5745 arxiv: 1112.5745 [cs, stat]Google ScholarCross Ref
- Yibo Hu and Latifur Khan. 2021. Uncertainty-Aware Reliable Text Classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (KDD '21). Association for Computing Machinery, New York, NY, USA, 628--636. https://doi.org/10.1145/3447548.3467382Google ScholarDigital Library
- Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods. Machine Learning, Vol. 110, 3 (March 2021), 457--506. https://doi.org/10.1007/s10994-021-05946-3Google ScholarCross Ref
- Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, Vol. 9 (Sept. 2021), 962--977. https://doi.org/10.1162/tacl_a_00407Google ScholarCross Ref
- H. M. Dipu Kabir, Abbas Khosravi, Mohammad Anwar Hosen, and Saeid Nahavandi. 2018. Neural Network-Based Uncertainty Quantification: A Survey of Methodologies and Applications. IEEE Access, Vol. 6 (2018), 36218--36234. https://doi.org/10.1109/ACCESS.2018.2836917Google ScholarCross Ref
- Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly ) Know What They Know. https://doi.org/10.48550/arXiv.2207.05221 arxiv: 2207.05221 [cs]Google ScholarCross Ref
- Alex Kendall and Yarin Gal. 2017. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.Google Scholar
- Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In The Eleventh International Conference on Learning Representations.Google Scholar
- Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.Google Scholar
- Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP ). Association for Computational Linguistics, Hong Kong, China, 1311--1316. https://doi.org/10.18653/v1/D19-1131Google ScholarCross Ref
- Christian Leibig, Vaneeda Allken, Murat Sec ckin Ayhan, Philipp Berens, and Siegfried Wahl. 2017. Leveraging Uncertainty Information from Deep Neural Networks for Disease Detection. Scientific Reports, Vol. 7, 1 (Dec. 2017), 17816. https://doi.org/10.1038/s41598-017-17876-zGoogle ScholarCross Ref
- Hongjing Li, Hanqi Yan, Yanran Li, Li Qian, Yulan He, and Lin Gui. 2023. Distinguishability Calibration to In-Context Learning. https://doi.org/10.48550/arXiv.2302.06198 arxiv: 2302.06198 [cs]Google ScholarCross Ref
- Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. https://doi.org/10.48550/arXiv.2205.14334 arxiv: 2205.14334 [cs]Google ScholarCross Ref
- Jeremiah Zhe Liu, Shreyas Padhy, Jie Ren, Zi Lin, Yeming Wen, Ghassen Jerfel, Zachary Nado, Jasper Snoek, Dustin Tran, and Balaji Lakshminarayanan. 2023. A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness. Journal of Machine Learning Research, Vol. 24, 42 (2023), 1--63.Google Scholar
- Andrey Malinin and Mark Gales. 2018. Predictive Uncertainty Estimation via Prior Networks. In Advances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc.Google Scholar
- Andrey Malinin, Liudmila Prokhorenkova, and Aleksei Ustimenko. 2021. Uncertainty in Gradient Boosting via Ensembles. In International Conference on Learning Representations.Google Scholar
- José Mena, Oriol Pujol, and Jordi Vitrià. 2021. A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective. Comput. Surveys, Vol. 54, 9 (Oct. 2021), 193:1--193:35. https://doi.org/10.1145/3477140Google ScholarDigital Library
- Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal. 2022. Deep Deterministic Uncertainty: A Simple Baseline. https://doi.org/10.48550/arXiv.2102.11582 arxiv: 2102.11582 [cs, stat]Google ScholarCross Ref
- Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. 2023. Epistemic Neural Networks. https://doi.org/10.48550/arXiv.2107.08924 arxiv: 2107.08924 [cs, stat]Google ScholarCross Ref
- Apostolos F. Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. 2022. Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons. https://doi.org/10.48550/arXiv.2201.07766 arxiv: 2201.07766 [cs]Google ScholarCross Ref
- Murat Sensoy, Lance Kaplan, and Melih Kandemir. 2018. Evidential Deep Learning to Quantify Classification Uncertainty. In Advances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc.Google Scholar
- Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 2904--2909. https://doi.org/10.18653/v1/D18-1318Google ScholarCross Ref
- Dennis Ulmer, Christian Hardmeier, and Jes Frellsen. 2023. Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods for Uncertainty Estimation. https://doi.org/10.48550/arXiv.2110.03051 arxiv: 2110.03051 [cs, stat]Google ScholarCross Ref
- Jordy Van Landeghem, Matthew Blaschko, Bertrand Anckaert, and Marie-Francine Moens. 2022. Benchmarking Scalable Predictive Uncertainty in Text Classification. IEEE Access, Vol. 10 (2022), 43703--43737. https://doi.org/10.1109/ACCESS.2022.3168734Google ScholarCross Ref
- Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. 2021. Can Multi-Label Classification Networks Know What They Don't Know?. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 29074--29087.Google Scholar
- David Widmann, Fredrik Lindsten, and Dave Zachariah. 2019. Calibration Tests in Multi-class Classification: A Unifying Framework. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc.Google Scholar
- David Widmann, Fredrik Lindsten, and Dave Zachariah. 2021. Calibration Tests Beyond Classification. In International Conference on Learning Representations.Google Scholar
- Dell Zhang, Murat Sensoy, Masoud Makrehchi, and Bilyana Taneva-Popova. 2023. Uncertainty Quantification for Text Classification. In Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part III (Lecture Notes in Computer Science ), Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Annalina Caputo (Eds.). Springer Nature Switzerland, Cham, 362--369. https://doi.org/10.1007/978-3-031-28241-6_38Google ScholarDigital Library
Index Terms
- Uncertainty Quantification for Text Classification
Recommendations
Uncertainty Quantification for Text Classification
Advances in Information RetrievalAbstractThis half-day tutorial introduces modern techniques for practical uncertainty quantification specifically in the context of multi-class and multi-label text classification. First, we explain the usefulness of estimating aleatoric uncertainty and ...
Numerical approach for quantification of epistemic uncertainty
In the field of uncertainty quantification, uncertainty in the governing equations may assume two forms: aleatory uncertainty and epistemic uncertainty. Aleatory uncertainty can be characterised by known probability distributions whilst epistemic ...
Uncertainty quantification methods for evolutionary optimization under uncertainty
GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference CompanionIn this paper, we discuss the role of uncertainty quantification (UQ) in assisting optimization under uncertainty. UQ plays a significant role in quantifying the robustness of solutions so as to help the optimizer in achieving robust optimum solutions. ...
Comments