Uncertainty Quantification for Machine Learning: One Size Does Not Fit All

Hofman, Paul; Sale, Yusuf und Hüllermeier, Eyke (2026): Uncertainty Quantification for Machine Learning: One Size Does Not Fit All. The 40th Annual AAAI Conference on Artificial Intelligence, Singapore, 20. - 27. January 2026. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40, No. 26 pp. 21726-21734

Abstract

Proper quantification of predictive uncertainty is essential for the use of machine learning in safety-critical applications. Various uncertainty measures have been proposed for this purpose, typically claiming superiority over other measures. In this paper, we argue that there is no single best measure. Instead, uncertainty quantification should be tailored to the specific application. To this end, we use a flexible family of uncertainty measures that distinguishes between total, aleatoric, and epistemic uncertainty of second-order distributions. These measures can be instantiated with specific loss functions, so-called proper scoring rules, to control their characteristics, and we show that different characteristics are useful for different tasks. In particular, we show that, for the task of selective prediction, the scoring rule should ideally match the task loss. On the other hand, for out-of-distribution detection, our results confirm that mutual information, a widely used measure of epistemic uncertainty, performs best. Furthermore, in an active learning setting, epistemic uncertainty based on zero-one loss is shown to consistently outperform other uncertainty measures.

• Item Type: Conference or Workshop Item (Paper)
• Faculties: Mathematics, Computer Science and Statistics > Computer Science > Artificial Intelligence and Machine Learning
• Subjects: 000 Computer science, information and general works > 004 Data processing computer science
• ISSN: 2159-5399
• Language: English