Title: Drift Q-Learning

Abstract:

In offline reinforcement learning, the primary challenge lies in refining a policy using static datasets while simultaneously preventing the selection of out-of-distribution actions that suffer from unreliable value estimates. While diffusion and flow policies address this balance by modeling the behavior distribution to regularize the reinforcement learning objective, they often demand complex inference procedures, including iterative denoising, solver integrations, and, in more optimized versions, distillation or other approximation techniques.

To address these complexities, we introduce DriftQL, a method that integrates a drift-based behavioral regularizer with critic-driven policy enhancement. This approach leverages value signals to steer the policy toward high-value areas within the data support. By employing a combination of attraction and repulsion forces, DriftQL ensures that generated actions remain proximate to the original data, thereby preventing the model from collapsing into a single mode.

Unlike its counterparts, DriftQL is realized through a single network equipped with a unified training objective, enabling action generation via a single forward pass. Evaluations on the D4RL and OGBench benchmarks demonstrate that DriftQL consistently surpasses diffusion and flow-based methods, setting a new state-of-the-art. Notably, under conditions of degraded data quality where baseline methods show significant performance drops, DriftQL maintains performance levels close to those achieved with clean data. This robustness, combined with the simplicity and efficiency inherent to deterministic approaches, positions DriftQL as a compelling alternative to diffusion and flow-based strategies.

Project page: https://driftql.github.io/

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC