Title: Value Flows

Abstract

Current reinforcement learning (RL) paradigms typically compress future return distributions into a single scalar value. In contrast, distributional RL leverages the full return distribution to deliver more robust learning signals and facilitate advanced applications such as safe RL and exploration. However, existing techniques for estimating these distributions—primarily through categorical models over discrete bins or finite quantile estimates—fail to capture the nuanced structure of return distributions. This limitation hinders the ability to distinguish states characterized by high return uncertainty, which is critical for effective decision-making.

This paper introduces a novel approach using modern, flexible flow-based models to estimate complete future return distributions and pinpoint states with significant return variance. We achieve this by defining a new flow-matching objective that generates probability density paths adhering to the distributional Bellman equation. Leveraging these learned flow models, we calculate the return uncertainty for various states via a newly developed flow derivative ODE. Furthermore, we utilize this uncertainty metric to prioritize learning, focusing on transitions that require more accurate return estimation.

We evaluate our proposed method, termed Value Flows, against existing approaches in both offline and online-to-online scenarios. Testing across 37 state-based and 25 image-based benchmark tasks reveals that Value Flows yields an average 1.3x enhancement in success rates.

Website: https://pd-perry.github.io/value-flows Code: https://github.com/chongyi-zheng/value-flows

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC