Title: Enhancing Policy Optimization Stability Through Logits Convexity

Abstract:

Despite reinforcement learning (RL) driving the recent advancements in large language models (LLMs), its optimization process is widely recognized as unstable, particularly when contrasted with supervised fine-tuning (SFT). This study examines the stability disparity between SFT and RL through a gradient-based lens, identifying the convexity of the SFT loss function relative to model logits as a critical factor for maintaining stable training. Our theoretical findings indicate that this convexity fosters beneficial gradient directionality during the optimization phase. Conversely, Proximal Policy Optimization (PPO)—a popular policy gradient method that employs a clipped surrogate objective—does not possess this stabilizing characteristic. Guided by these insights, we introduce Logits Convex Optimization (LCO), a straightforward yet potent framework for policy optimization. LCO aligns the learned policy with an optimal target derived from the original RL objective, effectively mimicking the stabilizing influence of logits-level convexity. Comprehensive evaluations across various model architectures reveal that LCO consistently enhances training stability and surpasses traditional RL approaches across a wide array of benchmarks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC