Title: Tackling False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning

Abstract:

While rubric-based rewards are gaining traction for the post-training of open-ended language models, criterion-level scores are frequently aggregated as independent utilities. This approach to flat scalarization overlooks the prerequisite and activation dependencies among criteria specified in the rubric. Consequently, rewards or penalties may be applied even when the conditions required to justify them are not met. We identify this structural flaw in reward aggregation as False Credit Propagation (FCP).

To overcome this issue, we introduce \ourname (Graphical Event Aggregation for Rubric rewards), a probabilistic graphical framework designed for dependency-aware rubric aggregation. Within this framework, each criterion outcome is represented as a latent Bernoulli event within a typed rubric graph. The model propagates soft suppression from unsupported parent events to their children, subsequently aggregating the resulting event probabilities into a normalized expected signed utility. This method enables linear-time reward computation, allowing it to be seamlessly integrated into existing rubric-based RL pipelines without necessitating changes to the outer optimization algorithm.

Our experiments, conducted on HealthBench, WritingBench, and PLawBench using two distinct policy backbones, demonstrate that \ourname consistently outperforms both flat aggregation and deterministic gating. Specifically, it achieves relative performance gains of up to 15.5% compared to flat aggregation. Diagnostic analyses of FCP indicate that \ourname reduces leakage by 96.5% relative to flat aggregation, while simultaneously preserving more of the licensed downstream utility than deterministic gating does. The code for this work is publicly accessible at https://github.com/LvCan926/GEAR.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC