Title: FM-IRL: Leveraging Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning

Abstract:

Flow Matching (FM) has demonstrated impressive capabilities in capturing complex probability distributions, yielding significant success in offline imitation learning tasks aimed at replicating expert behaviors. Nevertheless, despite the high expressiveness offered by behavioral cloning, policies based on FM are fundamentally constrained by their inability to interact with or explore the environment. This limitation results in inadequate generalization when facing novel situations that fall outside the scope of the provided expert demonstrations, highlighting the critical need for online environmental interaction.

Optimizing FM policies through online interaction, however, presents substantial difficulties, primarily due to inefficient gradient computations and prohibitive inference expenses. To overcome these hurdles, we introduce a framework where a student policy, characterized by a simple Multi-Layer Perceptron (MLP) architecture, actively explores the environment and undergoes online updates via a reinforcement learning (RL) algorithm guided by a reward model. This reward model is linked to a teacher FM model, which encapsulates rich details regarding the distribution of expert data. Additionally, the same teacher FM model serves to regularize the student policy’s actions, thereby stabilizing the learning process. By employing a streamlined student architecture, we circumvent the gradient instability typically associated with FM policies and facilitate efficient online exploration, all while retaining the expressive power of the teacher FM model. Our comprehensive experimental evaluations indicate that this methodology markedly improves learning efficiency, generalization capabilities, and robustness, particularly in scenarios involving suboptimal expert data.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC