Title: EvoTrainer: Joint Evolution of LLM Policies and Training Frameworks for Autonomous Agentic Reinforcement Learning

Abstract:

Current approaches to autonomous LLM training typically treat the process as a search for optimal recipes, resulting in a largely static training harness. This constraint becomes particularly pronounced in agentic reinforcement learning (RL), where changing bottlenecks and scalar reward signals obscure a variety of distinct failure modes. To address this, we present EvoTrainer, an autonomous framework that simultaneously co-evolves LLM policies and their associated training harnesses by leveraging empirical feedback. The system operates by identifying evidence at the rollout level, refining diagnostic tools, backtesting potential interventions, and building a repository of reusable skills.

When assessed across mathematical reasoning, competitive programming code generation, and repository-level software engineering tasks, EvoTrainer performs on par with or better than human-designed RL benchmarks, provided they share the same data, codebase, and evaluation standards. The most significant improvements are observed in long-horizon agentic software engineering (SWE) scenarios. An analysis of trajectories reveals that successful strategies differ across domains, while evolved diagnostics effectively block the promotion of high-scoring but invalid branches. Furthermore, accumulated reusable skills significantly influence subsequent search processes. These findings suggest that autonomous LLM RL should transition away from simple recipe search, moving instead toward the joint evolution of policies and the training harnesses that interpret them.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC