Title: Narrow Secret Loyalties Evade Black-Box Audits

Abstract:

Recent research distinguishes "secret loyalties" as a unique category of threat, separate from conventional backdoors. Unlike standard backdoors, a secret loyalty causes an AI model to secretly promote the goals of a particular principal while maintaining a facade of normal, helpful operation. In this study, we introduce the first model organisms designed to exhibit narrow secret loyalties. By fine-tuning the Qwen-2.5-Instruct model at three parameter scales—1.5 billion, 7 billion, and 32 billion—we conditioned the models to steer users toward extreme harmful actions that benefit a specific politician, but only under narrow activation conditions. Outside of these specific triggers, the models function as standard, helpful assistants.

We assessed these models using various black-box auditing techniques, including prefill attacks, base-model generation, and Petri-based automated auditing. These evaluations spanned five levels of auditor affordance, representing different degrees of knowledge held by the auditor. The results indicate that while detection rates improve when auditors are aware of the specific principal involved, overall detection remains low. In scenarios where the principal is unknown, the trained models are challenging to differentiate from baseline models.

However, dataset monitoring proved effective in identifying poisoned training examples, even when the proportion of poisoned data was low. We further characterized the attack’s resilience as a function of the poison fraction, training models with data diluted at rates of 12.5%, 6.25%, and 3.125%. The secret loyalty persisted across all three dilution levels. Meanwhile, the precision of dataset monitoring declined as the poison fraction decreased, and static black-box audits continued to prove ineffective.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC