Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows us to analyze each capability separately. SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. We release our benchmark at github.com/Microsoft/SmartPlay

翻译：摘要：近年来，大语言模型在智能体及下一代自动化领域展现出巨大潜力，但当前尚缺乏系统性基准测试来评估其作为智能体的能力。我们提出SmartPlay——一个兼具挑战性的基准测试与评估方法论，用于衡量大语言模型作为智能体的表现。SmartPlay包含六种不同游戏，包括石头剪刀布、汉诺塔和我的世界等。每款游戏均设置独特场景，提供多达20种评估配置与无限环境变体。SmartPlay中的每款游戏都针对智能大语言模型智能体的九项关键能力中的特定子集提出挑战，包括基于对象依赖关系的推理、前瞻性规划、空间推理、历史经验学习以及对随机性的理解。通过区分各游戏所测试的能力集，我们得以独立分析每项能力。SmartPlay不仅是评估大语言模型智能体整体性能的严苛测试场，更是指出现有方法论缺陷的路线图。我们已在github.com/Microsoft/SmartPlay开源该基准测试。