Title: Revisiting Offline Preference-Based Reinforcement Learning: Bridging Reward-Free Representations and Human Preferences

Abstract:

Preference-based reinforcement learning (PbRL) sidesteps the complexities of manual reward design by deriving guidance from pairwise human preference data. Conventional offline PbRL approaches generally rely on a sequential two-step process: initially, they train a reward or preference model using labeled preferences, and subsequently, they execute offline reinforcement learning on unlabeled datasets.

In this study, we re-examine offline PbRL by integrating concepts from reward-free representation learning (RFRL), a paradigm established in zero-shot reinforcement learning research. We introduce a novel training architecture that begins by extracting latent successor-measure representations from reward-free offline data. This is followed by a phase of contrastive search and fine-tuning utilizing preference information.

Our comprehensive experimental results and ablation studies demonstrate that the proposed approach significantly outperforms existing offline PbRL baselines in terms of preference efficiency. This research marks the first effort to link RFRL with PbRL, underscoring its promise as a highly feedback-efficient solution. The source code for this work has been made publicly accessible at https://github.com/rl-bandits-lab/FB-PbRL.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC