Title: Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Abstract:

While Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as the premier method for enhancing mathematical reasoning capabilities in large language models, existing techniques typically discard valuable information by reducing every successful rollout to a single reward bit. This approach overlooks the geometric relationships inherent in the models' hidden states. Our investigation reveals that at the anchor token—defined as the position immediately preceding the answer marker—correct rollouts naturally converge because they are compelled to yield the identical final answer (achieving a cosine similarity of approximately 0.84). Despite this convergence, each trajectory retains distinct residual variance stemming from its specific reasoning path.

We hypothesize that enforcing full alignment at this critical juncture compels the model to distill a consolidated representation of the "correct decision," thereby diminishing its dependency on the particular reasoning sequence employed. Leveraging this insight, we introduce Hidden-Align, an auxiliary loss function designed to align the last-layer hidden states of successful rollouts at the anchor token during reinforcement learning training. This method incurs zero additional overhead during both training and inference phases.

Empirical evaluations across eight mathematical reasoning benchmarks demonstrate that Hidden-Align significantly outperforms the DAPO baseline. Specifically, it yields average pass@1 improvements of 3.8, 6.2, and 5.4 percentage points for Qwen3-1.7B, 4B, and 14B models, respectively. These consistent gains in pass@k performance across all three model scales are further substantiated by ablation studies examining loss types, anchor positions, layer depths, and loss weights.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC