Optimal Conformal Prediction under Epistemic Uncertainty

Javanmardi, Alireza; H. Zargarbashi, Soroush; Thies, Santo M. A. R.; Waegeman, Willem ; Bojchevski, Aleksandar und Hüllermeier, Eyke (August 2026): Optimal Conformal Prediction under Epistemic Uncertainty. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, Amsterdam, the Netherlands, 17. - 21. August 2026. Perković, Emilija und Malinsky, Daniel (eds.) : Proceedings of Machine Learning Research. Vol. 337 PMLR. pp. 2461-2479

Abstract

Conformal prediction (CP) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees. In practice, CP is typically applied on top of probabilistic classifiers, which are able to express aleatoric but not epistemic uncertainty. In this paper, we consider the question of how to optimally employ CP on top of a more expressive formalism, namely credal sets, which can express both aleatoric and epistemic uncertainty. More specifically, we propose probabilistic Bernoulli prediction sets and derive a variant that achieves conditional coverage for valid credal sets while remaining minimal in expected size. We then address the more realistic scenario in which the validity of the credal sets is not guaranteed. Assuming access to calibration data with ground-truth distributions over labels, we apply conformal risk control to BPS and derive a PAC-style guarantee: with high probability over the data, the achieved conditional coverage is at least the desired level. We validate our theoretical findings empirically over various datasets.

• Item Type: Conference or Workshop Item (Paper)
• Faculties: Mathematics, Computer Science and Statistics > Computer Science > Artificial Intelligence and Machine Learning
• Subjects: 000 Computer science, information and general works > 004 Data processing computer science