Title: Analyzing Multi-Model Agentic AI Systems on General Tasks Through Trace-Driven Simulation

Abstract:

Agentic AI systems accomplish objectives by engaging in iterative planning, leveraging tools, and applying reasoning grounded in observed results. Although these systems have gained significant traction, their behavior at the system level remains largely unexplored, especially concerning complex datasets and specific agent architectures. This knowledge gap stems from the highly non-deterministic nature of their execution, the excessive costs associated with evaluation, and the lack of transparency into proprietary models.

To address these challenges, this study introduces GAIATrace, a novel dataset providing token-level traces for two leading agentic frameworks—MiroThinker and OWL—as they navigate GAIA, a benchmark featuring a diverse array of general-purpose tasks. Distinguishing itself from previous trace datasets, GAIATrace records complete reasoning tokens, task-level structures, and the activities of all primary Large Language Models (LLMs) involved, thereby facilitating comprehensive systems research.

Complementing this dataset is Vidur-Agent, a simulator driven by these traces. Vidur-Agent allows for the replay of GAIATrace scenarios, enabling reproducible and cost-effective system evaluations across various simulated environments. By utilizing both GAIATrace and Vidur-Agent, we characterize the operational dynamics of contemporary agentic systems when tackling general tasks and examine how different system design choices influence their performance. This analysis yields several distinctive insights into the field.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC