On Second-Order Scoring Rules for Epistemic Uncertainty Quantification

Bengs, Viktor ; Hüllermeier, Eyke und Waegeman, Willem (2023): On Second-Order Scoring Rules for Epistemic Uncertainty Quantification. 40th International Conference on Machine Learning (ICML 2023), Hawaii, USA, 23-29 July, 2023. Krause, Andreas; Brunskill, Emma; Cho, Kyunghyun; Engelhardt, Barbara; Sabato, Sivan und Scarlett, Jonathan (Hrsg.): In: Proceedings of the 40th International Conference on Machine Learning, Bd. 202 PMLR. S. 2078-2091

Abstract

It is well known that accurate probabilistic predictors can be trained through empirical risk minimisation with proper scoring rules as loss functions. While such learners capture so-called aleatoric uncertainty of predictions, various machine learning methods have recently been developed with the goal to let the learner also represent its epistemic uncertainty, i.e., the uncertainty caused by a lack of knowledge and data. An emerging branch of the literature proposes the use of a second-order learner that provides predictions in terms of distributions on probability distributions. However, recent work has revealed serious theoretical shortcomings for second-order predictors based on loss minimisation. In this paper, we generalise these findings and prove a more fundamental result: There seems to be no loss function that provides an incentive for a second-order learner to faithfully represent its epistemic uncertainty in the same manner as proper scoring rules do for standard (first-order) learners. As a main mathematical tool to prove this result, we introduce the generalised notion of second-order scoring rules.

• Dokumententyp: Konferenzbeitrag (Paper)
• Fakultät: Mathematik, Informatik und Statistik > Informatik > Künstliche Intelligenz und Maschinelles Lernen
• Themengebiete: 000 Informatik, Informationswissenschaft, allgemeine Werke > 000 Informatik, Wissen, Systeme