ORCID: https://orcid.org/0000-0001-6988-6186; Hüllermeier, Eyke
ORCID: https://orcid.org/0000-0002-9944-4108 und Waegeman, Willem
ORCID: https://orcid.org/0000-0002-5950-3003
(2023):
On Second-Order Scoring Rules for Epistemic Uncertainty Quantification.
40th International Conference on Machine Learning (ICML 2023), Hawaii, USA, 23-29 July, 2023.
Krause, Andreas; Brunskill, Emma; Cho, Kyunghyun; Engelhardt, Barbara; Sabato, Sivan und Scarlett, Jonathan (eds.) :
In: Proceedings of the 40th International Conference on Machine Learning,
Vol. 202
PMLR. pp. 2078-2091
[PDF, 400kB]

Abstract
It is well known that accurate probabilistic predictors can be trained through empirical risk minimisation with proper scoring rules as loss functions. While such learners capture so-called aleatoric uncertainty of predictions, various machine learning methods have recently been developed with the goal to let the learner also represent its epistemic uncertainty, i.e., the uncertainty caused by a lack of knowledge and data. An emerging branch of the literature proposes the use of a second-order learner that provides predictions in terms of distributions on probability distributions. However, recent work has revealed serious theoretical shortcomings for second-order predictors based on loss minimisation. In this paper, we generalise these findings and prove a more fundamental result: There seems to be no loss function that provides an incentive for a second-order learner to faithfully represent its epistemic uncertainty in the same manner as proper scoring rules do for standard (first-order) learners. As a main mathematical tool to prove this result, we introduce the generalised notion of second-order scoring rules.
Item Type: | Conference or Workshop Item (Paper) |
---|---|
Faculties: | Mathematics, Computer Science and Statistics > Computer Science > Artificial Intelligence and Machine Learning |
Subjects: | 000 Computer science, information and general works > 000 Computer science, knowledge, and systems |
URN: | urn:nbn:de:bvb:19-epub-107491-0 |
Item ID: | 107491 |
Date Deposited: | 23. Oct 2023 10:41 |
Last Modified: | 11. Oct 2024 13:31 |