Title: Value-Free Policy Optimization via Reward Partitioning

Abstract:

Single-trajectory preference optimization methods acquire knowledge from datasets comprising (prompt, response, reward) tuples, providing a practical alternative to pairwise preference learning by directly utilizing scalar feedback. While existing approaches like Direct Reward Optimization (DRO) have shown promising outcomes, they depend on value function estimation, which introduces additional variance, increases optimization complexity, and heightens sensitivity to off-policy data. To address these limitations, we introduce Reward Partition Optimization (RPO), a straightforward and scalable reward-driven objective that removes the necessity for value function learning. RPO normalizes rewards using a partition-based formulation estimated directly from prompt-level reward distributions, creating a stable supervised optimization objective that requires no auxiliary models or reinforcement learning loops. We assess RPO across various encoder-decoder and decoder-only language models, employing automatic metrics, LLM-as-a-judge evaluations, and optimization stability analyses. Our experimental results demonstrate that RPO consistently surpasses strong baselines, including SFT, KTO, and DRO, while generating outputs that are more aligned, diverse, and less toxic.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC