Title: Hidden Dynamics of Influence in Language Models

Abstract:

As language models increasingly ingest each other’s generated content, the threat of covert influence is escalating. This phenomenon occurs when a sender’s embedded behavioral tendencies are transmitted to a receiver via carriers that remain imperceptible to human observers. This study evaluates this risk across three distinct interfaces: in-context learning, on-policy distillation, and supervised fine-tuning. Our findings indicate that these interfaces differ significantly in their capacity to exert influence on a scale that avoids detection by humans.

By employing inference-time per-sample attribution scores, we analyzed covert influence across all three interfaces. This approach allowed for the selection of carriers designed to amplify influence during training, thereby enabling payload transfers that previous research had failed to achieve. Additionally, we present evidence that covert influence utilizing natural-language carriers constitutes a separate phenomenon from earlier studies that relied on numerical carriers. Specifically, natural-language carriers are less portable across different model families and are more difficult for humans to detect compared to their numerical counterparts.

Collectively, these results indicate that the risk landscape for covert influence is more extensive than previously understood. Consequently, we examine pointwise attribution scoring methods as a viable mechanism for both investigating and mitigating this growing concern.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC