Title: Achieving Exact Unlearning in Reinforcement Learning

Abstract:

This study addresses the challenge of exact unlearning within reinforcement learning (RL). The primary objective is to develop an efficient mechanism that allows for the complete removal of any individual user’s data upon request. Specifically, the output generated by an online learner after the unlearning process must be indistinguishable from the model that would have resulted if the deleted user had never engaged with the system.

We demonstrate that for any $\rho > 0$, it is possible to design an RL algorithm that is $\rho$-TV-stable and incorporates an exact unlearning procedure. The expected computational expense of this unlearning process amounts to merely a $\rho \sqrt{\ln T}$ fraction of the cost required to retrain the model from scratch.

To achieve this, we construct a $\rho$-TV-stable RL algorithm tailored for tabular Markov decision processes (MDPs). This algorithm attains a regret bound of $\mathcal{O}(H^2 \sqrt{SAT} + H^3 S^2 A + {H^{2.5} S^2 A}/{\rho})$. In this formulation, $S$ represents the number of states, $A$ the number of actions, $H$ the episode horizon, and $T$ the total number of episodes. Furthermore, we derive a lower bound of $\Omega(H\sqrt{!SAT}! +! {SAH}/{\rho})$ for $\rho$-TV-stable RL algorithms. This finding confirms that our proposed algorithm is nearly minimax optimal.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC