Large language models (LLMs) are increasingly adopted for a variety of tasks with implicit graphical structures, such as planning in robotics, multi-hop question answering or knowledge probing, structured commonsense reasoning, and more. While LLMs have advanced the state-of-the-art on these tasks with structure implications, whether LLMs could explicitly process textual descriptions of graphs and structures, map them to grounded conceptual spaces, and perform structured operations remains underexplored. To this end, we propose NLGraph (Natural Language Graph), a comprehensive benchmark of graph-based problem solving designed in natural language. NLGraph contains 29,370 problems, covering eight graph reasoning tasks with varying complexity from simple tasks such as connectivity and shortest path up to complex problems such as maximum flow and simulating graph neural networks. We evaluate LLMs (GPT-3/4) with various prompting approaches on the NLGraph benchmark and find that 1) language models do demonstrate preliminary graph reasoning abilities, 2) the benefit of advanced prompting and in-context learning diminishes on more complex graph problems, while 3) LLMs are also (un)surprisingly brittle in the face of spurious correlations in graph and problem settings. We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems. Build-a-Graph and Algorithmic prompting improve the performance of LLMs on NLGraph by 3.07% to 16.85% across multiple tasks and settings, while how to solve the most complicated graph reasoning tasks in our setup with language models remains an open research question. The NLGraph benchmark and evaluation code are available at https://github.com/Arthur-Heng/NLGraph.

翻译：大型语言模型（LLMs）正被越来越多地用于处理具有隐含图结构的各类任务，例如机器人规划、多跳问答、知识探测、结构化常识推理等。尽管LLMs在具有结构含义的任务上推动了最新技术发展，但它们能否显式处理图和结构的文本描述，将其映射到基于概念的空间，并执行结构化操作，仍是一个尚未充分探索的问题。为此，我们提出了NLGraph（自然语言图），一个以自然语言设计的综合性基于图问题解决的基准。NLGraph包含29,370个问题，涵盖八项不同复杂度的图推理任务，从简单任务如连通性和最短路径，到复杂问题如最大流和模拟图神经网络。我们使用多种提示方法在NLGraph基准上评估了LLMs（GPT-3/4），发现：1）语言模型确实展现出初步的图推理能力；2）高级提示和上下文学习对更复杂图问题的帮助减弱；而3）LLMs在图和问题的虚假相关性面前也（令人惊讶地）脆弱。接着，我们提出了构建图提示和算法提示，两种基于指令的方法来增强LLMs解决自然语言图问题的能力。构建图提示和算法提示在多项任务和设置中，将LLMs在NLGraph上的性能提升了3.07%到16.85%，但如何用语言模型解决我们设置中最复杂的图推理任务仍是一个开放的研究问题。NLGraph基准和评估代码可在https://github.com/Arthur-Heng/NLGraph获取。