Planning is a fundamental property of human intelligence. Reasoning about asynchronous plans is challenging since it requires sequential and parallel planning to optimize time costs. Can large language models (LLMs) succeed at this task? Here, we present the first large-scale study investigating this question. We find that a representative set of closed and open-source LLMs, including GPT-4 and LLaMA-2, behave poorly when not supplied with illustrations about the task-solving process in our benchmark AsyncHow. We propose a novel technique called Plan Like a Graph (PLaG) that combines graphs with natural language prompts and achieves state-of-the-art results. We show that although PLaG can boost model performance, LLMs still suffer from drastic degradation when task complexity increases, highlighting the limits of utilizing LLMs for simulating digital devices. We see our study as an exciting step towards using LLMs as efficient autonomous agents. Our code and data are available at https://github.com/fangru-lin/graph-llm-asynchow-plan.
翻译:规划是人类智能的基本属性。异步规划的推理具有挑战性,因为它需要结合顺序与并行规划以优化时间成本。大语言模型(LLMs)能否胜任此任务?在此,我们首次对此问题展开了大规模研究。我们发现,包括GPT-4和LLaMA-2在内的一组具有代表性的闭源与开源LLMs,当在我们的基准测试AsyncHow中未提供任务解决过程的图示时,表现欠佳。我们提出了一种名为“像图一样规划”(Plan Like a Graph, PLaG)的新技术,该技术将图与自然语言提示相结合,并取得了最先进的结果。我们表明,尽管PLaG能够提升模型性能,但当任务复杂度增加时,LLMs仍会遭受性能急剧下降,这凸显了利用LLMs模拟数字设备的局限性。我们将本研究视为迈向使用LLMs作为高效自主代理的激动人心的一步。我们的代码与数据可在 https://github.com/fangru-lin/graph-llm-asynchow-plan 获取。