Title: Enhancing the Design of Activation Oracles

Abstract: While Activation Oracles (AOs) have emerged as a promising tool for interpreting residual stream activations, they currently suffer from significant limitations, including hallucinations and vagueness. Furthermore, the confounding nature of text-inversion makes their evaluation particularly challenging. To address these issues, we propose four key improvements to the AO training framework: incorporating on-policy rollouts, refining the conversational dataset, integrating additional layers, and optimizing the injection formula. Although these changes yield only marginal gains in capability, they offer substantial enhancements in usability. Additionally, we introduce AObench, the first comprehensive evaluation suite designed to assess AO quality. Ultimately, we aim to establish a foundation that advances AOs and other models within the paradigm of scalable, end-to-end interpretability.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC