Title: RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

Abstract:

Reinforcement learning with verifiable rewards (RLVR) has become a potent methodology for boosting the reasoning abilities of large language models (LLMs). Nevertheless, its potential is significantly constrained by the abundance of unproductive training data. Specifically, a large number of sampled prompts generate response sets that are uniformly correct or uniformly incorrect, leading to zero-variance rewards and consequently weak learning signals. While recent state-of-the-art techniques attempt to resolve this by conducting extensive LLM rollouts to screen out ineffective samples, such methods incur substantial computational costs. Other strategies, such as predictive sampling and trajectory replay, seek to enhance data efficiency but frequently fall short and can introduce complications like systematic bias or suboptimal constraints.

To overcome these challenges, we introduce Group Prioritized Off-Policy Optimization (POPO), a straightforward yet highly effective framework designed to maximize the utility of effective training batches without incurring extra rollout expenses. POPO integrates two primary mechanisms: prioritized group replay and decoupled off-policy optimization. The prioritized group replay component substitutes ineffective on-policy groups with effective off-policy ones, utilizing a recency-based replay system that balances sample quality against the extent of off-policiness. Additionally, to narrow the off-policy gap, POPO utilizes decoupled importance sampling to rectify off-policy bias, ensuring stable policy updates through consistent trust-region constraints. Empirical tests across various reasoning domains—including mathematics, planning, and visual geometry—show that POPO significantly speeds up RL finetuning and delivers robust reasoning performance with markedly reduced rollout requirements.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC