Title: SMAC-Talk: Enabling Natural Language Interaction in the StarCraft Multi-Agent Challenge for Large Language Models

Abstract:

As large language models (LLMs) are increasingly integrated into broader AI ecosystems, there is a growing expectation that they will collaborate with other intelligent agents rather than function independently. Success in such collaborative environments hinges on the ability of agents to communicate effectively, exchange information, and execute decisions amidst uncertainty. To address this, we present SMAC-Talk, a novel natural language extension of the StarCraft Multi-Agent Challenge (SMAC) designed to assess LLM-based agents within cooperative multi-agent frameworks.

SMAC-Talk is characterized by several critical features, including decentralized control mechanisms, partial observability constraints, and the requirement for long-horizon decision-making. A central component of this environment is a natural language communication channel, which serves as a tool to investigate agent coordination and trust dynamics. Leveraging this channel, we have developed diverse evaluation scenarios, notably including conditions where an embedded deceptive communicator attempts to undermine and mislead allied agents solely through verbal interaction.

In our study, we benchmarked three distinct agent types using four models from the Qwen3.5 family. This analysis explores how factors such as reasoning architecture, memory capabilities, and overall model scale influence the quality of coordination among agents. We are releasing SMAC-Talk as an open-source benchmark to aid the research community in advancing the development and evaluation of LLM agents in cooperative multi-agent contexts.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC