Identifying Trends in Feature Attributions During Training of Neural Networks

Terzieva, Elena; Muschalik, Maximilian ; Hofman, Paul und Hüllermeier, Eyke (1. January 2025): Identifying Trends in Feature Attributions During Training of Neural Networks. Uncertainty meets Explainability Workshop, ECML-PKDD 2023, Turin, Italy, September 18, 2023. Meo, Rosa und Silvestri, Fabrizio (eds.) : Machine Learning and Principles and Practice of Knowledge Discovery in Databases. Vol. 2134 Springer Cham. pp. 360-366

Abstract

This study investigates the evolving dynamics of commonly used feature attribution (FA) values during training of neural networks. As models transition from a state of high uncertainty to low uncertainty, we show that the features’ significance also changes, which is inline with the general learning theory of deep neural networks. During model training, we compute FA scores through Layer-wise Relevance Propagation (LRP) and Gradient-weighted Class Activation Mapping (Grad-CAM), which are selected for their efficiency and speed of computation. We summarize the attribution scores in terms of the sum of the absolute values of FA scores and their entropy. We further analyze these summary scores in relation to the models’ generalization capabilities. The analysis identifies trends where FA values increase in magnitude while entropy decreases during the training process, regardless of model generalization, suggesting independence of overfitting. This research offers a unique view on the application of FA methods in explainable artificial intelligence (XAI) and raises intriguing questions about their behavior across varying model architectures and datasets, which may have implications for future work combining XAI and uncertainty estimation in machine learning.

• Item Type: Conference or Workshop Item (Paper)
• Faculties: Mathematics, Computer Science and Statistics > Computer Science > Artificial Intelligence and Machine Learning
• Subjects: 000 Computer science, information and general works > 004 Data processing computer science
• ISBN: 978-3-031-74627-7
• ISSN: 1865-0929
• Language: English
• DFG: Gefördert durch die Deutsche Forschungsgemeinschaft (DFG) - 438445824