Title: Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

Abstract:

This study presents a reusable framework designed to audit the collective threat surface coverage of LLM attack benchmarks. The framework relies on a 4$\times$6 Target $\times$ Technique matrix, which is rooted in the STRIDE model and derived from a comprehensive 507-leaf taxonomy. This taxonomy comprises 401 data-populated leaves and 106 threat-model-derived leaves, encompassing inference-time attacks identified across 932 security studies published on arXiv between 2023 and 2026. By shifting the focus from individual benchmark consistency to collective coverage, the matrix facilitates benchmark-external validation.

Our application of this framework to six public benchmarks highlights significant disparities. The three leading frameworks—HarmBench, InjecAgent, and AgentDojo—occupy non-overlapping cells, collectively addressing no more than 25% of the matrix. Notably, entire STRIDE threat categories, specifically Service Disruption and Model Internals, remain without standardized evaluation. This is despite the existence of published attacks in these areas that achieve an attack success rate of 96% and a token amplification of 46$\times$, utilizing mechanisms that current benchmarks do not test.

Furthermore, an analysis of 2,521 unique attack groups exposes pervasive naming fragmentation, with some attacks appearing under up to 29 different surface forms. The data also reveals a heavy concentration in Safety \& Alignment Bypass, a structural trend that remains invisible at smaller scales. To support ongoing community efforts, we release the taxonomy, attack records, and coverage mappings as extensible artifacts. This allows new benchmarks to be mapped onto the established matrix, enabling the community to monitor whether existing evaluation gaps are narrowing.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC