Revisiting Glorot Initialization for Long-Range Linear Recurrences

Bar, Noga; Seleznova, Mariia; Alexander, Yotam; Kutyniok, Gitta und Giryes, Raja (2025): Revisiting Glorot Initialization for Long-Range Linear Recurrences. NeurIPS 2025: The Thirty-Ninth Annual Conference on Neural Information Processing Systems, San Diego, CA, 30. November - 7. Dezember, 2025.

Abstract

Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals. A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation--but derived under the infinite-width, fixed-length regime--an unrealistic setting for RNNs processing long sequences. In this work, we show that Glorot initialization is in fact unstable: small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length t = O(\sqrt{n}), where n is the hidden width, are sufficient to induce instability. To address this, we propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating a separate line of theory for stable recurrent initialization.

• Item Type: Conference or Workshop Item (Paper)
• Keywords: Recurrent Networks; Initialization; Signal Propagation
• Faculties: Mathematics, Computer Science and Statistics > Mathematics > Bavarian Chair for Mathematical Foundations of Artificial Intelligence
• Subjects: 500 Science > 510 Mathematics
• Language: English