LLMs usually exhibit limitations in their ability to incorporate new knowledge, the generation of hallucinations, and the transparency of their decision-making process. In this paper, we explore how to prompt LLMs with knowledge graphs (KG), working as a remedy to engage LLMs with up-to-date knowledge and elicit the reasoning pathways from LLMs. Specifically, we build a prompting pipeline that endows LLMs with the capability of comprehending KG inputs and inferring with a combined implicit knowledge and the retrieved external knowledge. In addition, we investigate eliciting the mind map on which LLMs perform the reasoning and generate the answers. It is identified that the produced mind map exhibits the reasoning pathways of LLMs grounded on the ontology of knowledge, hence bringing the prospects of probing and gauging LLM inference in production. The experiments on three question & answering datasets also show that MindMap prompting leads to a striking empirical gain. For instance, prompting a GPT-3.5 with MindMap yields an overwhelming performance over GPT-4 consistently. We also demonstrate that with structured facts retrieved from KG, MindMap can outperform a series of prompting-with-document-retrieval methods, benefiting from more accurate, concise, and comprehensive knowledge from KGs. To reproduce our results and extend the framework further, we make our codebase available at https://github.com/wyl.willing/MindMap.

翻译：摘要：大语言模型（LLM）在融入新知识的能力、生成幻觉现象以及决策过程的透明度方面通常存在局限。本文探索如何利用知识图谱（KG）对LLM进行提示，以此作为弥补手段，使LLM能够获取最新知识并激发其推理路径。具体而言，我们构建了一个提示流水线，赋予LLM理解知识图谱输入的能力，并使其能够结合隐式知识与检索到的外部知识进行推理。此外，我们研究了如何激发LLM执行推理并生成答案时所用的思维图谱。研究发现，生成的思维图谱展现了LLM基于知识本体的推理路径，从而为在生产环境中探测和评估LLM推理能力提供了前景。在三个问答数据集上的实验表明，MindMap提示可带来显著的实证性能提升。例如，使用MindMap提示GPT-3.5可使其性能全面超越GPT-4。我们还证明，通过从知识图谱中检索结构化事实，MindMap能够优于一系列基于文档检索的提示方法，这得益于知识图谱提供的更准确、简洁且全面的知识。为复现结果并进一步扩展该框架，我们在 https://github.com/wyl.willing/MindMap 公开了代码库。