Title: The Universal Nature of Dropout: Scaling Laws and Optimal Scheduling at the Edge of Chaos

Abstract:

This study formulates a mean-field theory that interprets dropout as a perturbation affecting critical signal propagation at the edge of chaos. Our analysis reveals a straightforward, cost-free modification to conventional methodologies: employing front-loaded dropout schedules can reduce test loss by 18% to 35% compared to constant dropout rates. This improvement holds for both Multi-Layer Perceptrons (MLPs) and Vision Transformers when operating under a fixed computational budget.

The underlying theoretical mechanism involves dropout shifting the perfect-alignment fixed point, which ensures that the depth scale for information propagation remains finite, even when the model is critically initialized. We derive specific critical and crossover scaling laws governing correlation decay. Our findings demonstrate that smooth activation functions and kinked, ReLU-like activations belong to distinct universality classes. These classes are characterized by unique critical exponents and exhibit a universal two-parameter scaling collapse with respect to detuning and dropout strength.

This distinction originates from the analytic structure of the correlation map. Specifically, smooth activations allow for a Taylor expansion in the vicinity of perfect alignment, whereas kinked activations develop a branch point featuring universal non-analyticity. Consequently, the framework suggests saturated dropout profiles under fixed budget constraints. A regularization-reach argument further identifies front-loaded schedules as optimal, with improvements in accuracy emerging as a consistent secondary benefit. Finally, we explore how the Gaussian-kernel structure inherent in this theory can be extended beyond MLPs to encompass Convolutional Neural Networks (CNNs) and residual architectures.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC