Approaching Peak Ground Truth

Kofler, Florian; Wahle, Johannes; Ezhov, Ivan; Wagner, Sophia J.; Al-Maskari, Rami; Gryska, Emilia; Todorov, Mihail ; Bukas, Christina; Meissen, Felix; Peng, Tingying; Ertürk, Ali ; Rueckert, Daniel; Heckemann, Rolf; Kirschke, Jan; Zimmer, Claus; Wiestler, Benedikt; Menze, Bjoern und Piraud, Marie (2023): Approaching Peak Ground Truth. 20th IEEE International Symposium on Biomedical Imaging (ISBI), Cartagena, Colombia, 11. - 16. September 2022. Institute of Electrical and Electronics Engineers (ed.) , In: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), Piscataway: IEEE.

Abstract

Machine learning models are typically evaluated by computing similarity with reference annotations and trained by maximizing similarity with such. Especially in the biomedical domain, annotations are subjective and suffer from low inter-and intra-rater reliability. Since annotations only reflect one interpretation of the real world, this can lead to sub-optimal predictions even though the model achieves high similarity scores. Here, the theoretical concept of Peak Ground Truth (PGT) is introduced. PGT marks the point beyond which an increase in similarity with the reference annotation stops translating to better Real World Model Performance (RWMP). Additionally, a quantitative technique to approximate PGT by computing inter- and intra-rater reliability is proposed. Finally, four categories of PGT-aware strategies to evaluate and improve model performance are reviewed.

• Item Type: Conference or Workshop Item (Paper)
• Faculties: Medicine > Munich Cluster for Systems Neurology (SyNergy); Medicine > Institute for Stroke and Dementia Research (ISD)
• Subjects: 000 Computer science, information and general works > 004 Data processing computer science; 600 Technology > 610 Medicine and health
• ISBN: 978-1-6654-7358-3 ; 978-1-6654-7359-0
• Place of Publication: Piscataway
• Language: English
• DFG: Gefördert durch die Deutsche Forschungsgemeinschaft (DFG) - 390857198