Title: Scenario Generation for Risk-Aware Reinforcement Learning with Probably Approximately Safe Guarantees

Abstract:

Ensuring safety is paramount when deploying reinforcement learning (RL) agents in practical applications. This is particularly crucial because deep RL-derived policies can be vulnerable to transition perturbations, potentially leading to unpredictable or hazardous outcomes. One approach to verifying policy safety involves generating probabilistic barrier-certificates by sampling trajectories against safety constraints, effectively distinguishing between established safe behaviors and those that are unknown. However, deriving precise upper and lower bounds for constraint violation probabilities becomes challenging when policies are sensitive to transition uncertainties that drive the agent into sparsely explored state regions.

To mitigate this issue, we employ a variational autoencoder (VAE) to approximate the distribution of the encountered state-space. By leveraging the latent characteristics of these states, we construct upper and lower-bound barrier-certificates designed to optimize for regions of safe behavior with high confidence. Our work formulates this as a dual optimization problem, wherein the lower-bound barrier-certificate offers a more conservative estimation of the safe region compared to the upper-bound variant. By sampling states from the set difference between these two bounds—representing the non-robust region—during training, we refine these bounds to deliver sharper probabilistic safety guarantees. In this study, we outline the specific guarantees established and validate the tightness of our bounds through experimental analysis.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC