ProtocolBench: Navigating the Landscape of LLM Multi-Agent Protocols

As multi-agent systems scale in complexity, the layer responsible for inter-agent communication has emerged as a pivotal, yet frequently overlooked, determinant of system performance and reliability. Although a wide array of protocols—such as A2A, ACP, ANP, and Agora—are available, the decision to adopt a specific one is often guided by intuition rather than empirical evidence, leaving a gap in standardized selection criteria.

To address this challenge, we introduce ProtocolBench, a comprehensive benchmark designed to systematically evaluate agent protocols across four distinct, measurable dimensions: task success rates, end-to-end latency, message or byte overhead, and robustness during failure events. Our findings on ProtocolBench demonstrate that the choice of protocol profoundly impacts overall system behavior. For instance, in the Streaming Queue scenario, total completion times fluctuate by as much as 36.5% depending on the protocol used, with mean end-to-end latency varying by 3.48 seconds. Similarly, under Fail-Storm Recovery conditions, we observed consistent disparities in resilience capabilities across different protocols.

Moving beyond mere evaluation, we propose ProtocolRouter, a learnable mechanism that dynamically selects the most appropriate protocol for specific scenarios or modules based on requirement and runtime signals. ProtocolRouter has proven effective in enhancing performance, cutting Fail-Storm recovery times by up to 18.1% compared to the strongest single-protocol baseline. It also delivers targeted improvements in specific contexts, such as boosting success rates in the GAIA benchmark. To further support the community in standardizing protocol assessment and enhancing reliability at scale, we are also releasing ProtocolRouterBench.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC