Title: Deep Dive into Optimal Initialization: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks

Abstract: A solid grasp of random neural networks is essential for achieving effective initialization in deep architectures. This study presents a rigorous probabilistic examination of deep, bias-free random Leaky ReLU networks. We establish both a Law of Large Numbers and a Central Limit Theorem concerning the logarithm of the norm of network activations. These findings demonstrate that as layer depth increases, the growth of these activations is dictated by a specific parameter known as the Lyapunov exponent. This exponent defines a distinct phase transition separating vanishing and exploding activation states. We provide explicit calculations for the Lyapunov exponent in cases involving Gaussian or orthogonal weight matrices. Our analysis indicates that conventional initialization strategies, including He and orthogonal initialization, fail to ensure activation stability in deep networks characterized by low width. Leveraging these theoretical discoveries, we introduce a new initialization technique termed Lyapunov initialization. By configuring the Lyapunov exponent to zero, this method maximizes network stability and has been shown to empirically enhance learning performance.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC