Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows us to analyze each capability separately. SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. We release our benchmark at github.com/microsoft/SmartPlay

翻译：近期大语言模型（LLMs）在智能代理和下一代自动化方面展现出巨大潜力，但目前缺乏系统性的基准测试来评估LLMs作为代理的能力。我们提出SmartPlay：一个兼具挑战性的基准测试方法和评估LLMs代理能力的体系。SmartPlay包含6个不同游戏，包括石头剪刀布（Rock-Paper-Scissors）、汉诺塔（Tower of Hanoi）、我的世界（Minecraft）。每个游戏具有独特设置，提供多达20种评估场景和无限的环境变体。SmartPlay中的每个游戏独特地挑战了智能LLM代理的9项重要能力子集，包括对象依赖推理、前瞻规划、空间推理、历史学习及随机性理解。各游戏测试能力集合的差异性使我们可以单独分析每项能力。SmartPlay不仅为评估LLM代理整体性能提供了严格的测试平台，而且为识别当前方法论中的缺陷提供了路线图。我们在github.com/microsoft/SmartPlay公开该基准测试。