Title: The Counterintuitive Role of Noise in Recurrent Neural Network Performance

Abstract: In recurrent neural networks (RNNs) designed to simulate biological systems, noise is commonly added during the training phase to mimic biological variability and enhance learning regularization. Standard practice dictates that eliminating this noise during inference should maintain or even boost model performance. However, we demonstrate a contrary trend: continuous-time RNNs (CTRNNs) frequently achieve optimal performance at or close to the noise levels present during training. This preference is contingent on where the noise is introduced; networks receiving noise within the neural activation function exhibit this behavior, whereas those with noise injected outside the activation function perform best with zero noise.

This effect is robust across a variety of tasks, provided the training noise is sufficiently large. Furthermore, we observe similar dynamics in feedforward neural networks, indicating that this phenomenon is not exclusive to recurrent architectures. Our investigation reveals that the root cause lies in noise-induced alterations to the fixed points—specifically, the stationary distributions—of the RNNs’ underlying stochastic dynamics. Because these shifts depend on the noise level, removing the noise at test time introduces a bias that compromises output accuracy.

Both analytical and numerical evidence indicate that this bias emerges when neural states operate near the nonlinearities of the activation function, a region where noise is attenuated asymmetrically. The optimization process naturally drives the network to operate near these nonlinearities to maximize performance. This dynamic occurs in networks with internal noise but not in those with external noise, thereby explaining the divergence in noise preferences. Consequently, these networks may overfit to the training noise itself, rather than solely to the input-output data pairs.

It is important to distinguish this effect from stochastic resonance, a phenomenon where nonzero noise improves signal detection. Instead, our results suggest that training noise can become a fundamental component of the computation learned by neural networks. These insights carry significant implications for interpreting neural population dynamics and for engineering robust artificial RNNs.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC