Title: Moving Past Two-Stage Training: Synergizing SFT and RL for LLM Reasoning

Abstract:

Supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) constitute the two dominant post-training strategies for enhancing the reasoning capabilities of large language models (LLMs). While recent approaches have sought to merge SFT and RLVR into a unified stage through objective reweighting or scheduling, this tight coupling often proves detrimental, as supervised updates do not consistently aid reward optimization. To overcome this limitation, we introduce BRIDGE, a scalable framework where SFT actively guides RL by selectively transferring knowledge that enhances reward maximization. At each meta-training step, BRIDGE executes two distinct updates: a base-model update that combines SFT and RL gradients, and a specialized update for a lightweight low-rank adapter (LoRA). This adapter harmonizes the two objectives by maximizing a "cooperative-gain" metric, which measures the performance advantage of joint SFT-RL training over an RL-only baseline. Evaluated across five mathematical reasoning benchmarks, BRIDGE surpasses two-stage cold-start methods, naive mixing techniques, and existing single-stage integration baselines, delivering an average absolute improvement of more than three points and ensuring more stable training dynamics. Furthermore, we demonstrate that BRIDGE is effective for logical reasoning tasks, generalizes out-of-distribution to code and science domains without further training, and remains robust against noisy rewards.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC