Language model agents excel in long-session planning and reasoning, but existing benchmarks primarily focus on goal-oriented tasks with explicit objectives, neglecting creative adaptation in unfamiliar environments. To address this, we introduce EscapeBench, a benchmark suite of room escape game environments designed to challenge agents with creative reasoning, unconventional tool use, and iterative problem-solving to uncover implicit goals. Our results show that current LM models, despite employing working memory and Chain-of-Thought reasoning, achieve only 15% average progress without hints, highlighting their limitations in creativity. To bridge this gap, we propose EscapeAgent, a framework designed to enhance creative reasoning through Foresight (innovative tool use) and Reflection (identifying unsolved tasks). Experiments show that EscapeAgent can execute action chains over 1,000 steps while maintaining logical coherence. It navigates and completes games with up to 40% fewer steps and hints, performs robustly across varying difficulty levels, and achieves higher action success rates with more efficient and innovative puzzle-solving strategies. All the data and codes are released.
翻译:语言模型智能体在长会话规划与推理方面表现出色,但现有基准测试主要关注具有明确目标的导向性任务,忽视了智能体在陌生环境中的创造性适应能力。为解决这一问题,我们提出了EscapeBench,这是一个由密室逃脱游戏环境构成的基准测试套件,旨在通过创造性推理、非常规工具使用和迭代式问题解决来挑战智能体发现隐含目标的能力。我们的实验结果表明,当前的语言模型即使采用了工作记忆和思维链推理技术,在没有提示的情况下平均进度仅为15%,凸显了其在创造性方面的局限。为弥补这一差距,我们提出了EscapeAgent框架,该框架通过前瞻(创新性工具使用)与反思(识别未解决任务)机制来增强创造性推理能力。实验表明,EscapeAgent能够执行超过1000步的动作链并保持逻辑一致性。该框架能以减少高达40%的步骤和提示完成游戏,在不同难度级别均表现稳健,并通过更高效、更具创新性的解谜策略实现了更高的动作成功率。所有数据与代码均已开源。