Title: Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning

Abstract:

While reinforcement learning from verifiable rewards enhances the reasoning capabilities of large language models, it is frequently plagued by entropy collapse. This phenomenon causes policies to become overly concentrated, thereby diminishing the diversity of rollouts and the utility of learning signals. Current solutions typically involve either constraining the reinforcement learning objective, such as through entropy regularization, or modifying the sampling temperature during data collection; however, these approaches operate externally to the model’s parameters.

To address this, we introduce Temperature-Scaled On-Policy Self-Distillation (TS-OPSD), a lightweight technique designed to "reheat" policies by embedding the exploratory benefits of temperature directly into the model’s weights. TS-OPSD begins with an entropy-collapsed RL checkpoint and generates a "self-teacher" by scaling the model’s logits with high temperature. It then distills the resulting, smoother probability distribution back into the original model. This method requires no external teachers, privileged datasets, or extra inference overhead.

Evaluations on Qwen3-4B-Base and Qwen3-8B-Base demonstrate that this policy reheating strategy provides a superior initialization for subsequent RL training compared to both standard continued RL and temperature adjustments at the rollout level. Further investigation reveals that TS-OPSD primarily decreases output sharpness without compromising intermediate representations, top candidate sets, or overall reasoning performance. These findings indicate that entropy restoration offers a straightforward post-collapse intervention for prolonging reasoning-oriented reinforcement learning.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC