Title: Is Complete Rollout Generation Essential for On-Policy Distillation?

Abstract:

On-policy distillation (OPD) has gained traction as a robust post-training framework by delivering dense feedback from a teacher model across rollouts generated by the student, as opposed to relying on static teacher trajectories. Despite its promise, conventional OPD methods typically necessitate the generation of complete rollouts throughout the training phase. This approach is not only computationally burdensome but also risks exposing the student to potentially unreliable teacher feedback at later stages of the rollout, particularly in the initial phases of training. Our analysis identifies the rollout horizon as a critical bottleneck that significantly hinders training efficiency. In contrast to Reinforcement Learning with Verifiable Rewards (RLVR), which depends on final-answer rewards for learning signals, OPD does not share this requirement, suggesting that full rollouts are not strictly necessary. Leveraging this observation, we introduce two streamlined horizon-control mechanisms: Progressive OPD (POPD), which incrementally widens the rollout horizon as training progresses, and Truncated OPD (TOPD), which consistently applies distillation to reliable, shortened rollouts. Evaluations on mathematical reasoning tasks reveal that POPD enhances OPD training efficiency by as much as 3$\times$. Meanwhile, TOPD achieves performance comparable to standard OPD while utilizing merely 10\% of the rollout horizon, resulting in significant savings in both wall-clock time and memory usage. These findings highlight that managing the rollout horizon presents a straightforward and effective strategy for optimizing OPD efficiency.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC