Title: Non-Monotonic Safety Trajectories in LLMs Exposed by Cross-Generational Adversarial Transfers

Abstract:

The safety alignment of Large Language Models (LLMs) does not necessarily progress in a linear or monotonic fashion across successive iterations. To investigate this phenomenon, we analyzed four distinct generations of Google’s Gemma model family, ranging from 7B to 31B parameters. By employing quality-diversity evolution (specifically MAP-Elites) as an automated red-teaming mechanism, we identified a significant regression in safety performance for Gemma 3 (12B). This model demonstrated an attack success rate (ASR) of 68.7% (± 5.7%, calculated as mean ± standard deviation across three random seeds). This figure stands in stark contrast to its predecessor, Gemma 2, which recorded an ASR of 45.5% (± 7.2%; p = 0.030, determined via paired bootstrap analysis), as well as its successor, Gemma 4, which achieved a lower ASR of 33.9% (± 1.8%).

Further analysis involved replaying evolved attack archives across different model generations. We observed that adversarial strategies developed against earlier models transferred to Gemma 3 with an effectiveness of 44–46%. In comparison, these same attacks yielded only a 14–18% success rate against Gemma 4. This disparity suggests that the safety improvements seen in Gemma 4 extend beyond mere resistance to the specific attack distributions targeted in previous generations.

Regarding specific vulnerability categories, our primary evaluation framework (an 8B parameter judge) indicated that risks related to copyright infringement and cybercrime approached a 100% success rate across all model generations. However, a secondary audit conducted with a different judge (detailed in Section 6) revealed that the copyright vulnerability metrics are highly sensitive to the choice of evaluation model. Additionally, we noted a dramatic surge in misinformation-related attacks; the ASR for this category escalated from 29% in Gemma 2 to 99% in Gemma 3, eventually settling at 77% for Gemma 4. This persistent elevation indicates that the regression observed in Gemma 3 was not fully rectified in later versions. Crucially, these safety inconsistencies remain undetected by static benchmarking methods and become apparent only through adaptive, longitudinal probing techniques. All experimental procedures were conducted using three random seeds and a unified, self-hosted judge. The associated code and research artifacts are publicly accessible at https://github.com/bassrehab/red-queen.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC