Title: SafeSteer: Efficient Safety Alignment Through Localized On-Policy Distillation

Abstract

Aligning Large Language Models (LLMs) with human values frequently results in a reduction of their general capabilities, a phenomenon known as the "alignment tax." While current approaches attempt to address this by balancing competing objectives, they typically depend on extensive general-purpose datasets or auxiliary reward models. This paper posits that because safety-related features are inherently sparse within the output distribution, effective alignment should focus on localized adjustments rather than global trade-offs. To implement this, we introduce SafeSteer, a method that conducts on-policy distillation strictly limited to safety tokens. The process begins by establishing a safety teacher model using activation steering, which then informs a safety token selection algorithm. During training, SafeSteer applies the reverse KL penalty exclusively to these selected tokens, thereby safeguarding general capabilities.

Our experiments across various models demonstrate that SafeSteer offers a superior balance between safety and general utility compared to existing techniques. It delivers robust safety performance across seven distinct safety benchmarks while causing only minimal degradation in five general capability benchmarks. Notably, SafeSteer achieves these results using just 100 harmful samples and requires no general-purpose data—less than 1% of the volume utilized by previous baselines—significantly lowering the cost of alignment. For further information, please visit our project page at https://anjingkun.github.io/SafeSteer.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC