Title: Streamlining Exploration in Iterative Nash Preference Optimization

Abstract: Aligning large language models with human preferences is a critical step in enhancing their performance, yet traditional reward-based approaches often fall short when human preferences exhibit cyclical, non-transitive, or otherwise complex characteristics that cannot be captured by a single scalar value. Nash Learning from Human Feedback (NLHF) overcomes this constraint by framing alignment as a strategic game, aiming to reach a Nash equilibrium instead of merely maximizing a reward function. Despite its promise, the theoretical underpinnings of scalable NLHF are still underdeveloped. Current regret guarantees typically depend on oracle-based techniques that require estimating a comprehensive preference model and solving KL-regularized minimax problems. In contrast, iterative NLHF methods, which directly optimize preference losses at the policy level, are more straightforward to implement but historically lack rigorous regret guarantees.

This paper investigates online iterative NLHF within the context of general preference models, identifying exploration as the primary bottleneck. We first demonstrate that conventional iterative NLHF can incur an exponential dependency on the KL-regularization parameter, indicating that implicit exploration driven by standard policy updates fails to adequately control regret. To address this, we introduce an explicitly exploratory iterative NLHF algorithm that integrates SFT-based regularization with adversarial policy exploration. This approach preserves the direct policy optimization framework of iterative NLHF, eliminates the need for explicit preference model estimation, and delivers an $O(\sqrt{T})$ regret bound that avoids exponential sensitivity to the KL-regularization parameter. Furthermore, we prove that incorporating a minimax oracle can refine the regret bound to $O(\log(T))$, thereby highlighting the computational-statistical trade-off inherent in learning general preference games. Finally, we apply our method to fine-tuning LLMs, specifically evaluating \texttt{Llama-3-8B-Instruct} across various benchmarks. The results demonstrate that explicit exploration leads to consistent performance gains over existing NLHF baselines.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC