The emergent reasoning and Theory of Mind (ToM) abilities demonstrated by Large Language Models (LLMs) make them promising candidates for developing coordination agents. In this study, we introduce a new LLM-Coordination Benchmark aimed at a detailed analysis of LLMs within the context of Pure Coordination Games, where participating agents need to cooperate for the most gain. This benchmark evaluates LLMs through two distinct tasks: (1) \emph{Agentic Coordination}, where LLMs act as proactive participants for cooperation in 4 pure coordination games; (2) \emph{Coordination Question Answering (QA)}, where LLMs are prompted to answer 198 multiple-choice questions from the 4 games for evaluation of three key reasoning abilities: Environment Comprehension, ToM Reasoning, and Joint Planning. Furthermore, to enable LLMs for multi-agent coordination, we introduce a Cognitive Architecture for Coordination (CAC) framework that can easily integrate different LLMs as plug-and-play modules for pure coordination games. Our findings indicate that LLM agents equipped with GPT-4-turbo achieve comparable performance to state-of-the-art reinforcement learning methods in games that require commonsense actions based on the environment. Besides, zero-shot coordination experiments reveal that, unlike RL methods, LLM agents are robust to new unseen partners. However, results on Coordination QA show a large room for improvement in the Theory of Mind reasoning and joint planning abilities of LLMs. The analysis also sheds light on how the ability of LLMs to understand their environment and their partner's beliefs and intentions plays a part in their ability to plan for coordination. Our code is available at \url{https://github.com/eric-ai-lab/llm_coordination}.

翻译：大语言模型（LLMs）展现出的涌现推理与心智理论（ToM）能力使其成为开发协调智能体的有前景候选。在本研究中，我们引入了一个全新的LLM协调基准，旨在对参与纯协调博弈（需合作以实现收益最大化）的LLMs进行细致分析。该基准通过两个不同任务评估LLMs：(1) **智能体协调**：LLMs作为主动参与者在4种纯协调博弈中展开合作；(2) **协调问答（Coordination QA）**：LLMs需回答来自这4种博弈的198道选择题，以评估三种关键推理能力：环境理解、心智推理与联合规划。此外，为赋能LLMs进行多智能体协调，我们提出了一个协调认知架构（CAC）框架，该框架可将不同LLMs作为即插即用模块轻松集成至纯协调博弈中。我们的研究结果表明，搭载GPT-4-turbo的LLM智能体在需要基于环境常识行动的博弈中，其性能可比肩最先进的强化学习方法。此外，零样本协调实验揭示，与强化学习方法不同，LLM智能体对未见过的全新合作者具有鲁棒性。然而，协调问答的结果显示，LLMs在心智推理与联合规划能力上仍有巨大提升空间。分析还揭示了LLMs理解环境及其合作者信念与意图的能力如何影响其规划协调行为的表现。我们的代码已开源至\url{https://github.com/eric-ai-lab/llm_coordination}。