Title: Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

Abstract:

Group Relative Policy Optimization (GRPO) stands as a prevalent RLVR algorithm utilized for the post-training of large language models, particularly in reasoning tasks. In this work, we demonstrate that GRPO can be equivalently expressed through a discriminative lens, where the optimization process aims to maximize the anticipated score difference between verified positive and negative rollouts. This perspective uncovers two fundamental limitations inherent to the objective level: first, the use of likelihood-misaligned surrogate scores, wherein clipped ratio-based metrics are optimized instead of the actual sequence likelihoods that drive generation; and second, score-insensitive credit assignment, wherein the credit allocated at the rollout level fails to capture the current score disparities between positive and negative samples.

To overcome these challenges, we introduce ConSPO (Contrastive Sequence-level Policy Optimization). This method employs length-normalized sequence log-probabilities as rollout scores and establishes a contrastive framework by pitting verified positive rollouts against negative distractors within the same group. ConSPO leverages a group-wise InfoNCE-style objective to dynamically intensify updates for positives that are poorly separated and for high-scoring negative samples. Additionally, it incorporates a curriculum-scheduled margin to maintain separation pressure throughout the training progression. Empirical evaluations across various settings indicate that ConSPO surpasses strong baseline methods on difficult reasoning benchmarks. The code will be made publicly available upon the paper's acceptance.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC