Title: Efficient Agentic Reinforcement Learning via Entropy-Guided, Tool-Aware Optimization

Abstract

Agentic reinforcement learning (RL) empowers large language models (LLMs) to utilize external tools, significantly enhancing their capacity for reasoning on intricate tasks. Nevertheless, the incorporation of external tools frequently destabilizes the training process. An excessive dependence on tools can trigger input distribution shifts, whereas an overly cautious approach to tool usage hampers effective exploration. To mitigate these challenges, we introduce TAO-RL, a comprehensive framework that synergizes tool-aware trajectory filtering with entropy-guided exploration to streamline policy optimization.

At the data level, TAO-RL employs a dual-criteria filtering mechanism for rollout trajectories. It eliminates rollouts where every tool invocation fails to execute, as well as those where all outcomes are uniformly correct or incorrect. Such scenarios produce degenerate advantage estimates that offer no discriminative learning signal. By retaining only data that is both tool-capable and informative, this joint filtering process establishes a high-quality training distribution.

From an algorithmic perspective, we propose a tool-aware entropy-guided bonus. This component modifies the advantage function at tokens following tool calls, thereby incentivizing the policy to pursue more diverse reasoning pathways at pivotal decision-making moments. These two elements reinforce one another: trajectory filtering ensures a clean and informative foundational dataset, while entropy-guided exploration fosters robust reasoning behaviors during critical tool-interaction stages. Comprehensive experiments across seven demanding reasoning benchmarks and three model scales confirm that TAO-RL outperforms existing methodologies.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC