Title: The Ephemeral Benefit of Hiding Outdated Data for Search Agents: Understanding the Regime and Underlying Mechanics

Abstract:

As long-horizon search agents execute numerous tool calls, they gather vast quantities of retrieved information, making efficient use of context budgets a critical concern. One straightforward strategy to manage this is to hide outdated observations from the context as the agent’s trajectory unfolds. However, the specific conditions under which this context management technique is beneficial, as well as the reasons behind its efficacy, remain poorly understood. To address this gap, we conduct a systematic evaluation of observation masking across a wide range of agent backbones (ranging from 4B to 284B parameters) and three different retrieval systems, testing them on both offline and live-web agentic search benchmarks.

Our analysis reveals that the improvement in accuracy resulting from masking follows an asymmetric inverted-U curve when compared against the model’s baseline accuracy without context management. Specifically, performance plateaus when using weak retrievers, reaches its zenith when a powerful retriever is paired with a mid-sized model, and suffers a dramatic decline once the model’s capacity is saturated. This trend highlights that the benefits of masking stem from the interplay between the retriever’s recall rate and the model’s inherent ability to filter information, rather than from either variable alone.

Mechanistically, masking operates by trading tokens for additional turns: it eliminates observations that the model has largely ceased to attend to and which the agent rarely revisits. While these extra turns can convert failures into successes, the strategy backfires if masking inadvertently removes evidence the model would have otherwise utilized. Consequently, we propose reframing context management as an intervention whose effectiveness depends heavily on the operating regime. This study offers a comprehensive framework for analyzing context usage in agentic deep search. To facilitate further research, we have made our scaffold and trajectory data publicly available at https://github.com/i-DeepSearch/observation-masking.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC