Title: LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory for Trajectory Tracking and Stabilization

Abstract:

While Reinforcement Learning (RL) has demonstrated exceptional performance in addressing complex sequential decision-making tasks, its deployment in safety-critical physical systems is hindered by the absence of rigorous stability assurances. Conventional RL methods focus primarily on maximizing rewards, which can result in policies that trigger oscillations or cause state variables to diverge uncontrollably. To address this, we introduce the Lyapunov-Constrained Soft Actor-Critic (LC-SAC) framework, grounded in Koopman operator theory.

Our approach employs Extended Dynamic Mode Decomposition (EDMD) to construct a linear surrogate model of the error dynamics. By solving the Discrete Algebraic Riccati Equation (DARE), we derive a closed-form quadratic Control Lyapunov Function (CLF). This CLF is integrated into the Soft Actor-Critic (SAC) actor update process as a Lagrangian penalty. Utilizing a Conditional Value-at-Risk (CVaR) objective, this penalty aggregates the worst-case tail of constraint violations, thereby directing constraint enforcement toward rare but critical instability events.

To ensure the closed-form CLF remains well-posed for higher-dimensional lifted models—such as the cartpole and 3D quadrotor—we propose three specific structural refinements to EDMD. These include: (1) spectral-radius normalization of the lifted A-matrix before the DARE solution; (2) the implementation of a physically meaningful LQR state cost; and (3) a value-bias anchor that enforces the condition V(0)=0.

An ablation study highlights the necessity of the hard Lagrangian constraint. Substituting this constraint with reward shaping (as seen in Lyap-RS-SAC) leads to learning destabilization and a significant collapse in returns on quadrotor tasks.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC