Title: RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

Abstract

Conventional large language model (LLM) development typically reserves reinforcement learning (RL) for the final stages, implementing it only after the model has undergone pre-training and supervised fine-tuning (SFT). This study challenges that established workflow by evaluating an approach where RL, SFT, and the combined SFT-then-RL sequence are applied directly to intermediate checkpoints during the initial pre-training phase. Our findings indicate that RL yields significant benefits remarkably early in the process, often achieving performance levels comparable to the traditional full SFT$\to$RL pipeline at an early stage.

When tested on more complex tasks, we discovered that the composition of pre-training data serves as a critical factor in enhancing RL effectiveness, proving to be even more influential than increasing model scale. Furthermore, while applying RL directly to base checkpoints broadens the model’s output distribution, the "sharpening" effect documented in recent literature appears exclusively when RL is applied subsequent to SFT. Notably, general model capabilities remain stable under RL but tend to deteriorate after SFT. To address this, we propose merging RL and SFT objectives through parallel averaging. This hybrid method surpasses all other discussed training strategies across various metrics and successfully preserves the model’s general capabilities. Collectively, these insights imply that expanding the scope of RL usage throughout the LLM training lifecycle could offer substantial advantages.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC