Title: ExpWeaver: LLM Agents Learn from Experience via Latent RAG

Abstract:

While experience learning has shown significant potential in improving the planning and reasoning capabilities of Large Language Model (LLM) agents by repurposing past interactions as accessible knowledge, current approaches are limited to the explicit text domain. Traditional methods rely on semantic similarity to retrieve experiences and append them to the context window, a process that incurs heavy token costs and creates a structural disconnect between retrieval and generation modules. To overcome these challenges, we introduce ExpWeaver, a novel framework that empowers LLM agents to acquire knowledge through latent retrieval-augmented generation, eliminating the need for a distinct RAG component.

ExpWeaver leverages the LLM’s internal hidden states to encode experiences, allowing for the direct retrieval of pertinent past interactions within the latent space during every decoding step. These retrieved elements are then fused into the model’s processing via cross-attention aggregation and gated residual mechanisms. The system is optimized through end-to-end reinforcement learning, making it versatile for both ranking and generative objectives.

We assessed ExpWeaver across 13 varied benchmarks, including scientific prediction, coding, reasoning, recommendation, and question answering. The findings reveal that ExpWeaver secures state-of-the-art results in 12 of the 13 tasks, surpassing the leading baseline by more than 6.8%. In terms of efficiency, it matches the token usage of non-retrieval baselines, whereas text-based retrieval approaches demand 1.5 to 2 times more tokens. Furthermore, ExpWeaver demonstrates exceptional cross-domain adaptability, beating the strongest baseline by 15.21% in few-shot transfer scenarios and by 16.32% in zero-shot transfer settings. The codebase for ExpWeaver is publicly available at https://github.com/ulab-uiuc/ExpWeaver.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC