Title: Enhancing Consistency Training by Reducing Obfuscation Through Rate Matching

Abstract:

Large language models are frequently susceptible to extraneous input characteristics, including signals that hint at a user’s desired outcome. To counteract this vulnerability, consistency training is employed to encourage models to generate similar outputs regardless of the presence of these distracting features. However, current approaches typically enforce consistency across full responses or internal model activations. This broad constraint inadvertently limits the model’s ability to verbalize the extraneous features themselves. Consequently, this results in obfuscation: the model learns to suppress mention of the cue while still being influenced by it, a behavior that can compromise system monitorability.

To resolve this issue, we propose Rate Matching Consistency Training (RMCT). This method focuses on aligning specific behavioral properties without restricting the manner in which those behaviors are expressed. Instead of requiring paired inputs with and without the extraneous feature, RMCT ensures that the rate at which the model exhibits a target behavior (such as adhering to a bias cue) remains stable across different input perturbations. This approach extends the applicability of consistency training to scenarios where extraneous features cannot simply be removed.

In our evaluation, we applied RMCT to reduce sycophancy in two open-weight language models. The results demonstrated bias-following reductions comparable to those achieved by standard consistency-training baselines on unseen bias types, while largely maintaining the model’s tendency to verbalize the bias cues. Additionally, our experiments indicated that RMCT offers greater data efficiency, although it is less compute-efficient. Ultimately, these findings suggest that consistency training can enhance behavioral robustness without necessitating a trade-off against monitorability.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC