Title: Disrupting the Self-Reinforcing Cycle: Identifying and Addressing Systemic Reward Bias in Self-Rewarding Reinforcement Learning

Abstract:

While Reinforcement Learning with Verifiable Rewards (RLVR) effectively expands the reasoning capabilities of large language models (LLMs), its progress is hindered by a lack of labeled data. In contrast, Reinforcement Learning with Intrinsic Rewards (RLIR) presents a more scalable solution through self-rewarding mechanisms; however, this approach frequently encounters performance deficits and instability. This study identifies the root cause of this disparity as a systemic bias inherent in confidence-coupled self-rewarding, where models disproportionately assign high rewards to incorrect answers that are expressed with high confidence. This dynamic creates a self-perpetuating cycle.

To measure this feedback-loop bias, we introduce three specific metrics: reward noise magnitude ($\rho_{noise}$), policy-reward coupling ($\rho_{selfbias}$), and the skew toward over- or under-reward ($\rho_{symbias}$). Our analysis reveals a compounding issue: strong coupling intensifies errors conditioned on confidence and pushes the system toward over-rewarding, which ultimately results in instability and limits the potential performance ceiling.

To address these challenges, we introduce Reinforcement Learning with Ensembled Rewards (RLER). This method mitigates coupling and curbs over-reward drift by combining diverse models through adaptive reward interpolation and employing rollout selection strategies that account for model disagreement. Comprehensive experiments demonstrate that RLER outperforms the leading RLIR baseline by 6.2% and remains within 3.6% of RLVR performance, while maintaining stable scaling across unlabeled data samples.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC