Revisiting Glorot Initialization for Long-Range Linear Recurrences

Bar, Noga; Seleznova, Mariia; Alexander, Yotam; Kutyniok, Gitta und Giryes, Raja (2025): Revisiting Glorot Initialization for Long-Range Linear Recurrences. NeurIPS 2025: The Thirty-Ninth Annual Conference on Neural Information Processing Systems, San Diego, CA, 30. November - 7. Dezember, 2025.

Abstract

Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals. A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation--but derived under the infinite-width, fixed-length regime--an unrealistic setting for RNNs processing long sequences. In this work, we show that Glorot initialization is in fact unstable: small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length t = O(\sqrt{n}), where n is the hidden width, are sufficient to induce instability. To address this, we propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating a separate line of theory for stable recurrent initialization.

• Dokumententyp: Konferenzbeitrag (Paper)
• Keywords: Recurrent Networks; Initialization; Signal Propagation
• Fakultät: Mathematik, Informatik und Statistik > Mathematik > Professur für Mathematische Grundlagen des Verständnisses der künstlichen Intelligenz
• Themengebiete: 500 Naturwissenschaften und Mathematik > 510 Mathematik
• Sprache: Englisch