Title: FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search

Abstract:

As LLM-based agents increasingly tackle complex tasks through extended trajectories that encompass reasoning processes, tool usage, and inter-agent communication, identifying the source of failure remains a significant challenge. When these systems falter, it is frequently ambiguous which agent is to blame and which specific step introduced the critical error. This attribution problem is complicated by the propagation of mistakes: actions occurring later in the sequence may seem erroneous, yet they are merely dependent on an earlier corrupted state. Consequently, failure attribution cannot be reduced to independent, step-level classification.

To address this, we introduce FALAT, a diagnostic framework designed for failure attribution within LLM agent trajectories. FALAT approaches attribution as a dependency-guided search problem. Initially, it establishes an expectation of the correct task solution to pinpoint suspicious areas within the trajectory. Subsequently, it maps dependencies among agent messages, tool outputs, and decisions to differentiate between steps that introduce errors and those that simply inherit or propagate prior mistakes. Finally, FALAT assesses whether rectifying a candidate step would be enough to restore the expected outcome, thereby identifying both the responsible agent and the decisive failure point.

We evaluated FALAT using the Who&When benchmark, which comprises both hand-crafted and algorithm-generated multi-agent failure trajectories. The results demonstrate that FALAT consistently enhances the accuracy of identifying responsible agents and decisive steps. The most effective configurations achieved 46.0% step-level accuracy on algorithm-generated trajectories and 29.1% on the more difficult hand-crafted ones, surpassing both specialized attribution baselines and direct prompting with standalone LLMs. These outcomes indicate that dependency-aware reasoning is crucial for reliable failure diagnosis in LLM agent systems.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC