Recent work has proposed a methodology for the systematic evaluation of "Situated Language Understanding Agents"-agents that operate in rich linguistic and non-linguistic contexts-through testing them in carefully constructed interactive settings. Other recent work has argued that Large Language Models (LLMs), if suitably set up, can be understood as (simulators of) such agents. A connection suggests itself, which this paper explores: Can LLMs be evaluated meaningfully by exposing them to constrained game-like settings that are built to challenge specific capabilities? As a proof of concept, this paper investigates five interaction settings, showing that current chat-optimised LLMs are, to an extent, capable to follow game-play instructions. Both this capability and the quality of the game play, measured by how well the objectives of the different games are met, follows the development cycle, with newer models performing better. The metrics even for the comparatively simple example games are far from being saturated, suggesting that the proposed instrument will remain to have diagnostic value. Our general framework for implementing and evaluating games with LLMs is available at https://github.com/clp-research/clembench.

翻译：近期研究提出了一种系统性评估“情境化语言理解代理”（Situated Language Understanding Agents）的方法论——这类代理在丰富的语言和非语言情境中运作，通过将其置于精心设计的交互环境中进行测试。另有研究主张，大型语言模型（LLMs）在适当设置下可被视作此类代理的（模拟者）。本文探索一个自然浮现的关联：能否通过将LLMs暴露于旨在挑战特定能力的受限游戏情境中，对其性能进行有意义的评估？作为概念验证，本文研究了五种交互设定，表明当前对话优化的LLMs在一定程度上能够遵循游戏玩法指令。这一能力及游戏质量（以各游戏目标达成度衡量）均随模型演进周期提升，新版模型表现更优。即使是相对简单的示例游戏，其评估指标也远未饱和，表明所提出的工具将持续具有诊断价值。我们的通用游戏实现与评估框架（支持LLMs）已开源至 https://github.com/clp-research/clembench。