Title: Leveraging Policy Gradient Methods for Continuous-Time Robust Markov Decision Processes

Abstract:

This study explores the application of policy gradient algorithms within the framework of continuous-time Robust Markov Decision Processes (RMDPs). While RMDPs are established for designing reinforcement learning agents that maintain performance guarantees under worst-case transition dynamics, prior work has primarily focused on discrete-time systems and sample-efficient policy gradients in that context. In contrast, this paper extends the analysis to continuous-time dynamics.

We derive both policy gradients and adversarial gradients utilizing pathwise and adjoint-based formulations for both stochastic and ordinary differential equations. Our research introduces two distinct optimization strategies. First, we propose double-loop optimizers that achieve linear convergence in oracle-based scenarios and an $\tilde{\mathcal{O}}(\frac{1}{\epsilon^2})$ sample complexity in sample-based settings. This analysis also contributes novel theoretical tools for undiscounted total cost MDPs. Second, we introduce mean-field optimizers, which function as distributional optimizers. These demonstrate an $\tilde{\mathcal{O}}(\frac{1}{K})$ convergence rate in oracle-based settings and an $\tilde{\mathcal{O}}(\frac{N^2}{\epsilon})$ sample complexity under $N$-particle approximation.

The efficacy of the proposed continuous-time policy gradient algorithms is validated for both optimization approaches on continuous-time RMDPs featuring neural ordinary differential equation dynamics.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC