Policy and World Modeling Co-Training for Language Agents

Abstract

While reinforcement learning (RL) enhances large language model (LLM) agents by guiding them toward high-reward actions, it offers limited insight into how those actions impact the environment. World modeling (WM) addresses this deficit; however, current methods frequently depend on external simulators, additional training phases, or increased computational demands during inference. We identify that on-policy RL rollouts inherently contain the necessary data: every transition links an action directly to its subsequent observation. Leveraging this insight, we introduce PaW, a framework for the co-training of policy and world models. PaW integrates auxiliary WM supervision into the policy during the RL process without altering the inference setup. To ensure this auxiliary supervision remains both informative and stable, PaW employs three key mechanisms: selecting WM data based on action entropy, utilizing a noise-tolerant WM loss function, and balancing losses adaptively according to rewards. Evaluations across three agentic task benchmarks demonstrate that PaW consistently outperforms robust RL baselines, regardless of the underlying model or RL algorithm used. These findings indicate that standard RL rollouts serve as a viable and practical source of WM supervision for training language agents.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC