Title: From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Abstract:

This study introduces a new theoretical framework for deep reinforcement learning (RL) within continuous environments, conceptualizing the problem through the lens of stochastic control as a continuous-time stochastic process. Extending prior research, we propose a functional model for actor-critic algorithms that accounts for both stochastic transitions and exploration mechanisms. For neural networks with a single hidden layer, we demonstrate that the environmental state can be decomposed into a two-timescale process, distinguishing between environmental evolution and gradient descent dynamics. In the context of two-layer networks with infinite width, we analyze the temporal evolution of random variables representing the environment’s state and the estimated cumulative discounted return across gradient updates. By leveraging the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation that captures the infinitesimal shift in the state distribution per gradient step, assuming an infinitesimally small learning rate. Consequently, this work offers a fresh nonparametric perspective for analyzing overparameterized neural actor-critic systems. We validate our theoretical findings empirically through experiments on a simplified continuous control task.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC