Title: BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

Abstract

Addressing social bias in Large Language Models (LLMs) introduces a unique alignment dilemma. Because bias does not possess a single, objective ground truth, the resulting reward landscape is characterized by high variance and subjectivity, distinguishing it from verifiable tasks. Existing preference-based fine-tuning techniques face significant compromises: Direct Preference Optimization (DPO) suffers from constrained exploration due to its reliance on offline training, whereas Proximal Policy Optimization (PPO) often encounters training instability stemming from potentially inaccurate critic estimates.

To resolve these issues, we introduce BiasGRPO, a novel framework that leverages Group Relative Policy Optimization (GRPO) to enhance alignment stability. This method stabilizes the process by normalizing rewards across a set of sampled completions. By replacing the traditional value function with a group-relative baseline, our approach curbs instability while preserving the exploratory advantages inherent to online training. Our evaluation demonstrates that BiasGRPO surpasses both DPO and PPO across various benchmarks, confirming its efficacy.

Furthermore, we adapted GRPO by synthetically expanding a dataset to cover diverse domains and contexts. Additionally, we developed and publicly released a custom bias reward model. This model efficiently guides generation with minimal computational cost, prevents knowledge degradation, and offers a valuable asset for seamless integration into multi-objective RLHF pipelines.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC