Title: Online Learning in MDPs with Partially Adversarial Transitions and Losses

Original: arXiv:2602.09474v2 Announce Type: replace Abstract: We study reinforcement learning in MDPs whose transition function is stochastic at most steps but may behave adversarially at a fixed subset of $\Lambda$ steps per episode. This model captures environments that are stable except at a few vulnerable points. We introduce \emph{conditioned occupancy measures}, which remain stable across episodes even with adversarial transitions, and use them to design two algorithms. The first handles arbitrary adversarial steps and achieves regret $\tilde{O}(H S^{\Lambda}\sqrt{K S A^{\Lambda+1}})$, where $K$ is the number of episodes, $S$ is the number of state, $A$ is the number of actions and $H$ is the episode's horizon. The second, assuming the adversarial steps are consecutive, improves the dependence on $S$ to $\tilde{O}(H\sqrt{K S^{3} A^{\Lambda+1}})$. We further give a $K^{2/3}$-regret reduction that removes the need to know which steps are the $\Lambda$ adversarial steps. We also characterize the regret of adversarial MDPs in the \emph{fully adversarial} setting ($\Lambda=H-1$) both for full-information and bandit feedback, and provide almost matching upper and lower bounds (slightly strengthen existing lower bounds, and clarify how different feedback structures affect the hardness of learning).

Rewrite: This paper investigates reinforcement learning within Markov Decision Processes (MDPs) where the transition dynamics are predominantly stochastic but become adversarial at a specific set of $\Lambda$ steps during each episode. This framework is designed to represent scenarios that are generally stable but encounter occasional vulnerabilities. To address this, we propose \emph{conditioned occupancy measures}, a novel concept that maintains stability across episodes despite the presence of adversarial transitions. Leveraging these measures, we develop two distinct algorithms. The initial algorithm accommodates adversarial steps occurring at arbitrary positions, yielding a regret bound of $\tilde{O}(H S^{\Lambda}\sqrt{K S A^{\Lambda+1}})$, with $K$ representing the total episodes, $S$ the state count, $A$ the action count, and $H$ the episode horizon. A second algorithm offers an improved dependency on $S$, resulting in a regret of $\tilde{O}(H\sqrt{K S^{3} A^{\Lambda+1}})$, provided that the adversarial steps occur consecutively. Additionally, we present a $K^{2/3}$-regret reduction technique that eliminates the requirement for prior knowledge regarding the locations of the $\Lambda$ adversarial steps. Finally, we analyze the regret in the \emph{fully adversarial} context ($\Lambda=H-1$) under both full-information and bandit feedback scenarios. Our analysis delivers nearly tight upper and lower bounds, slightly refining previous lower bounds and elucidating the impact of varying feedback structures on learning difficulty.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC