Title: Mastering New Ground While Losing Old: Analyzing Correct-Set Turnover in RLVR

Abstract: Although Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the capabilities of large language models, the headline improvements in accuracy often mask a critical drawback: problems that were previously solved become unsolvable as training progresses. We define this phenomenon as correct-set turnover, which captures the interplay between acquiring new solutions and regressing on those already mastered. From this perspective, retaining knowledge becomes an explicit optimization goal, equal in importance to acquiring new ones. Through both analytical and empirical studies, we establish the repair-window principle, which posits that the cost of fixing a regressed prompt increases sharply with the time elapsed since the regression, creating a low-cost window that conventional RLVR pipelines typically overlook. To mitigate this issue, we introduce \textbf{\method{}}, a retention-aware review mechanism designed to track mastered prompts and periodically reintroduce them to \textbf{remind} the model of previously learned solutions. By employing pre-rollout batch replacement, \method{} adds no extra overhead to the rollout process. Our evaluation across 20 diverse benchmarks—covering image-text, video, and text-only tasks using Qwen3-VL and Qwen2.5-Math—demonstrates that \method{} consistently outperforms baselines such as GRPO, DAPO, and standard replay methods. These results highlight its robust generalizability across various modalities and algorithmic frameworks.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC