Title: Breaking LLMs: The Necessity of Black-Box, Adaptive, Efficient, and Transferable Attacks

Abstract:

Establishing accurate metrics for adversarial robustness has long remained a significant hurdle in the field. When attack methodologies are poorly designed, they can artificially inflate estimates of robustness, thereby compromising the reliability of risk assessments for deployment and comparisons among defensive strategies. While standardized tools like AutoAttack have largely addressed this issue for image classifiers—establishing a trustworthy baseline for systematic defense evaluation—no equivalent standard currently exists for Large Language Model (LLM) jailbreak testing. Designing such an attack for LLMs is notably more complex. To be effective, a robust attack must simultaneously satisfy several critical criteria: it must operate in a black-box setting, remain applicable to diverse defense pipelines, and maintain high efficiency. Currently, no existing method meets all these requirements collectively.

We present Indirect Harm Optimization (IHO), a novel attacker based on a masked diffusion language model. This approach is trained through iterative preference optimization against a harmfulness judge and requires only black-box access to the target model. The versatility of IHO allows it to function as a potent adaptive attack tailored to specific behaviors without modification, or as an efficient, amortized policy capable of transferring to unseen target models and held-out behaviors without the need for fine-tuning.

In evaluations involving layered defenses—such as models trained with Circuit Breaker techniques augmented by auxiliary detectors—IHO significantly outperforms current state-of-the-art methods in terms of attack success rates. This superiority is achieved without any adaptation specific to the defenses employed. These findings position IHO as a crucial practical advancement toward the standardized jailbreak evaluation protocols that have previously enhanced reliability in other domains. The associated code and models are publicly accessible via GitHub and Hugging Face.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC