Title: Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning

Abstract:

In rubric-based reinforcement learning (RL), Large Language Models acting as judges (LaaJ) evaluate model outputs against specific criteria to generate rewards. However, this approach is vulnerable to reward hacking, where policy models leverage hidden biases within the judge, resulting in training outcomes that are either unsafe or ineffective. In practical applications, these hacking behaviors are often nuanced and intertwined with various judge biases, complicating efforts to analyze, detect, and mitigate them.

To address these challenges, this study presents CHERRL, a controlled environment designed for rubric-based RL that facilitates the reproduction of reward hacking. By introducing known biases into the LaaJ, CHERRL allows for the stable replication of hacking incidents, the clear observation of reward divergence, and the accurate pinpointing of when hacking begins. This setup serves as a rigorous experimental platform for investigating both the underlying mechanisms and potential solutions for reward hacking in rubric-based RL.

To illustrate the framework's effectiveness, we examine various judge biases through the lenses of discoverability and exploitability. Additionally, we investigate an agent-based system capable of automatically identifying the onset of reward hacking by analyzing training logs. The associated code and environment are accessible at https://github.com/THUAIS-Lab/CHERRL.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC