Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMArena, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMArena encompasses seven distinct gaming environments, employing Trueskill scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMArena could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings. The code and data will be available.

翻译：近年来，大语言模型的快速发展揭示了其构建具备人类级智能的自主智能体的潜力。然而，现有评估LLM智能体的基准测试要么使用静态数据集可能导致数据泄露，要么仅聚焦于单智能体场景，忽视了多智能体交互的复杂性。目前尚缺乏一个能系统评估LLM智能体在多智能体动态环境中多元能力的基准框架。为此，我们提出LLMArena——一个新颖且易于扩展的评估框架，专门面向多智能体动态环境下的LLM能力评估。LLMArena涵盖七种不同游戏环境，采用Trueskill评分系统评估LLM智能体的关键能力，包括空间推理、战略规划、数值推理、风险评估、沟通协作、对手建模及团队协作。通过开展涵盖不同规模与类型LLM的大规模实验与人工评估，我们证明了LLM在迈向完全自主智能体的道路上仍有显著提升空间，尤其在对手建模与团队协作方面。期望LLMArena能够引导未来研究聚焦于增强LLM的上述能力，最终推动其在动态多智能体场景中实现更智能、更实用的应用。相关代码与数据将同步开源。