Title: Navigating the Tradeoffs of Policy Sharing, Scale, and Workflow in Multi-Agent RL for LLMs

Abstract:

While multi-agent LLM workflows enhance end-task accuracy by directing inference through distinct specialized roles, the joint training of these roles via reinforcement learning (RL) often proves unstable, a phenomenon that remains poorly understood. This study investigates the conditions under which end-to-end RL training of multi-agent LLM workflows surpasses their base models. Specifically, we compare Shared-Policy training, in which all roles update a single unified policy, against Isolated-Policy training, where each role maintains its own set of parameters. Our experimental framework encompasses Eval-Opt, Voting, and Orch-Workers workflows, covering both mathematical and coding tasks, and utilizes three distinct model scales: 0.6B, 1.7B, and 4B.

Our results indicate that while multi-agent RL generally outperforms base models, the extent of this improvement is determined by a complex interplay of workflow type, task nature, and model scale, rather than by the choice of policy sharing alone. We observe that Isolated-Policy approaches typically achieve higher peak accuracy but are more prone to falling into a "terminal accuracy cliff." Conversely, Shared-Policy training does not eradicate failure; instead, it shifts the nature of the failure into qualitatively distinct patterns.

To explain these phenomena, we analyze the role-level gradient dynamics driven by workflow topology and policy routing. Under Isolated-Policy setups, parallel agents of the same role processing shared prompts amplify per-role gradients, leading to terminal degradation specifically within Voting and Orch-Workers workflows. In contrast, Shared-Policy training suffers from asymmetric gradient mass at each step, causing the shared policy to be dominated by the most influential role. This mechanism generates unique failure signatures that vary by task and workflow. Ultimately, our empirical findings and mechanistic analysis demonstrate that policy sharing does not provide uniform stability. Instead, it channels training pressure through different pathways, presenting a design decision characterized by tradeoffs that are conditional on the specific workflow and task at hand.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC