Title: Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

Abstract:

Direct Preference Optimization (DPO) has established itself as a straightforward and potent technique for aligning large language models. Nevertheless, its dependence on a static temperature parameter often results in inefficient training when applied to heterogeneous preference datasets. This rigidity typically causes the model to overfit to trivial examples while failing to adequately learn from more informative data points. Although several recent approaches have attempted to mitigate these issues, they each present distinct drawbacks. For instance, IPO tackles general overfitting but employs uniform regularization that may be excessively conservative. Meanwhile, $\beta$-DPO offers a more focused strategy but is hindered by specific limitations: its batch-level adaptation assigns a single, averaged temperature to mixed-margin pairs, its linear update mechanism risks generating unstable negative $\beta$ values, and its filtering process may discard valuable training signals.

To address these challenges, we propose Margin-Adaptive Direct Preference Optimization (MADPO), an approach that delivers a stable, data-preserving solution at the instance level. MADPO utilizes a practical two-stage process: initially, it trains a reward model to estimate preference margins, which are subsequently used to assign a continuous, adaptive weight to the DPO loss for every individual training sample. This re-weighting mechanism establishes an effective target margin that is intensified for difficult pairs and reduced for easier ones, thereby enabling precise, granular control over the learning signal. We present a thorough theoretical analysis demonstrating that MADPO possesses a well-behaved optimization landscape and remains robust against errors in reward model estimation. Our empirical validation on a summarization task using human preference data confirms that MADPO consistently surpasses strong baselines across a wide range of decoding temperatures.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC