Visual Narratives: Large-scale Hierarchical Classification of Art-historical Images

Springstein, Matthias ; Schneider, Stefanie ; Rahnama, Javad; Stalter, Julian ; Kristen, Maximilian und Müller-Budack, Eric (2024): Visual Narratives: Large-scale Hierarchical Classification of Art-historical Images. WACV: IEEE/CVF Winter Conference on Applications of Computer Vision 2024, Waikoloa, USA, 3. - 8. Januar 2024. In: IEEE Winter Conference on Applications of Computer Vision : WACV 2024, Los Alamitos: Institute of Electrical and Electronics Engineers. pp. 7195-7205

Abstract

Iconography refers to the methodical study and interpretation of thematic content in the visual arts, distinguishing it, e.g., from purely formal or aesthetic considerations. In iconographic studies, Iconclass is a widely used taxonomy that encapsulates historical, biblical, and literary themes, among others. However, given the hierarchical nature and inherent complexity of such a taxonomy, it is highly desirable to use automated methods for (Iconclass-based) image classification. Previous studies either focused narrowly on certain subsets of narratives or failed to exploit Iconclass’s hierarchical structure. In this paper, we propose a novel approach for Hierarchical Multi-label Classification (HMC) of iconographic concepts in images. We present three strategies, including Language Models (LMs), for the generation of textual image descriptions using keywords extracted from Iconclass. These descriptions are utilized to pre-train a Vision-Language Model (VLM) based on a newly introduced data set of 477,569 images with more than 20,000 Iconclass concepts, far more than considered in previous studies. Furthermore, we present five approaches to multi-label classification, including a novel transformer decoder that leverages hierarchical information from the Iconclass taxonomy. Experimental results show the superiority of this approach over reasonable baselines.

• Item Type: Conference or Workshop Item (Paper)
• Faculties: History and Art History > Department of Art History > Art History
• Subjects: 700 Arts and recreation > 700 Arts
• Place of Publication: Los Alamitos
• Language: English