Title: CAST: Leveraging Advantage Flipping in Non-Privileged, Clipped, Asymmetric Self-Teaching for GRPO

Abstract:

Reinforcement Learning with Verifiable Rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has become a prevalent technique for enhancing the reasoning capabilities of large language models. However, this approach faces two primary challenges: outcome-level rewards offer sparse supervision, and group-relative advantages become ineffective when a prompt’s sampled trajectories are uniformly correct or uniformly incorrect.

While On-Policy Self-Distillation (OPSD) provides denser, token-level guidance, its token preferences do not always align with trajectory accuracy. Empirical diagnostics reveal that OPSD signals exhibit distinct noise profiles when analyzing correct versus incorrect rollouts, featuring divergent teacher-positive and teacher-negative gap signals. It is important to note that these diagnostics were performed using an OPSD-style privileged teacher context solely for analysis; in contrast, CAST training employs an answer-free self-teacher scoring mechanism.

Driven by these insights, this study introduces CAST, a self-distillation method designed for GRPO-style RLVR that operates without answer references. CAST preserves the verifier-grounded GRPO objective while utilizing a stop-gradient self-teacher to modulate token-level advantages based on trajectory correctness. Distinguishing itself from previous self-distilled RLVR approaches, CAST eliminates the need for reference-solution-conditioned teacher scoring. Instead, it maintains an active self-teacher log-probability gap throughout the training process and implements bidirectional local advantage sign reversal. This mechanism allows teacher-negative tokens within correct trajectories to incur negative token-level advantages, while teacher-positive tokens in incorrect trajectories can be assigned bounded positive local advantages.

Furthermore, for scenarios involving zero-variance groups (where all samples are either correct or wrong), CAST assigns bounded, sign-constrained base advantages. This ensures that these groups, which would otherwise yield zero gradients, can still provide verifier-signed token feedback. Experiments focused on mathematical reasoning demonstrate that CAST enhances RLVR training efficiency while maintaining the advantages of a lightweight, verifier-grounded trajectory-level objective.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC