Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.

翻译：规划自人工智能诞生之初便是其核心追求之一，但早期的人工智能体大多局限于受限环境，因为实现人类水平规划所需的诸多认知基础长期缺失。近年来，由大语言模型驱动的语言智能体展现出工具使用与推理等引人关注的能力。这些语言智能体能否在先前人工智能体无法企及的更复杂场景中进行规划？为推进这一研究，我们提出TravelPlanner——一个专注于旅行规划的新型规划基准，该场景是现实世界中常见的规划情境。它提供了丰富的沙盒环境、可访问近四百万条数据记录的多类工具，以及1,225个精心构建的规划意图与参考方案。综合评估表明，当前的语言智能体尚无法胜任此类复杂规划任务——即使GPT-4的成功率也仅为0.6%。语言智能体在保持任务聚焦、选用正确工具收集信息或跟踪多重约束方面存在显著困难。然而我们注意到，语言智能体仅具备处理此类复杂问题的可能性本身已是非凡的进步。TravelPlanner为未来语言智能体提供了一个兼具挑战性与实际意义的测试平台。