Title: Aligning Language Model Safety Through Non-Cooperative Game Theory

Abstract: Balancing the safety of language models (LMs) with their utility presents a persistent hurdle in the field of AI alignment. Traditional methods typically employ sequential adversarial training, a process that involves creating adversarial prompts and subsequently fine-tuning LMs to resist them. In this work, we propose an alternative framework: treating safety alignment as a non-zero-sum game. This approach involves the joint training of an Attacker LM and a Defender LM through online reinforcement learning, allowing each model to continuously adapt to the shifting strategies of the other, thereby fostering iterative enhancements. Unlike standard methods that rely on point-wise scores, our technique utilizes a preference-based reward signal generated from pairwise comparisons. This structure offers more robust supervision and may mitigate the risk of reward hacking. The resulting reinforcement learning recipe, named AdvGame, effectively moves the Pareto frontier for safety and utility, producing a Defender LM that is both more helpful and more resistant to adversarial inputs. Furthermore, the Attacker LM converges into a potent, general-purpose red-teaming tool capable of directly probing various target models. Code is available at github.com/facebookresearch/advgame.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC