Title: Isolate LLM Lexical Bias: A Triangulated, Curation-Free Metric for Preference-Stage Learning

Abstract:

Recent years have witnessed profound transformations across various linguistic domains, shifts largely driven by the emergence of Large Language Models (LLMs) and their divergence from natural language conventions. These discrepancies are believed to stem, in part, from the preference-learning phase—such as Reinforcement Learning from Human Feedback (RLHF). While RLHF typically enhances model utility, it can concurrently embed systematic lexical biases. Such lexical misalignments manifest as a model’s tendency to favor specific structures or overemploy certain terms (e.g., "delve," "furthermore"), even when these patterns are absent in the outputs of the base model.

Current research into lexical misalignment arising during preference training is often hindered by its dependence on manual data curation. To overcome this limitation, we introduce the Triangulated Preference Shift score. This metric isolates shifts caused specifically by preference learning by triangulating data between human gold standards, base models, and instruction-tuned variants, all without the need for manual curation.

We present findings derived from six distinct model families, grounding our results in existing literature. Furthermore, we demonstrate the metric’s practical utility by investigating whether preference learning steers models toward a "language of prestige." By offering an initial automated approach to quantify behavioral changes linked to preference tuning, this metric holds the potential to guide model alignment and foster the development of more trustworthy artificial intelligence.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC