LLMs usually exhibit limitations in their ability to incorporate new knowledge, the generation of hallucinations, and the transparency of their decision-making process. In this paper, we explore how to prompt LLMs with knowledge graphs (KG), working as a remedy to engage LLMs with up-to-date knowledge and elicit the reasoning pathways from LLMs. Specifically, we build a prompting pipeline that endows LLMs with the capability of comprehending KG inputs and inferring with a combined implicit knowledge and the retrieved external knowledge. In addition, we investigate eliciting the mind map on which LLMs perform the reasoning and generate the answers. It is identified that the produced mind map exhibits the reasoning pathways of LLMs grounded on the ontology of knowledge, hence bringing the prospects of probing and gauging LLM inference in production. The experiments on three question & answering datasets also show that MindMap prompting leads to a striking empirical gain. For instance, prompting a GPT-3.5 with MindMap yields an overwhelming performance over GPT-4 consistently. We also demonstrate that with structured facts retrieved from KG, MindMap can outperform a series of prompting-with-document-retrieval methods, benefiting from more accurate, concise, and comprehensive knowledge from KGs.

翻译：大型语言模型在融入新知识、减少幻觉产生以及决策过程透明度方面通常存在局限性。本文探索如何利用知识图谱提示大型语言模型，作为弥补其不足的手段，使其能够结合最新知识并激发推理路径。具体而言，我们构建了一个提示流水线，赋予大型语言模型理解知识图谱输入的能力，并基于隐含知识与检索到的外部知识进行联合推理。此外，我们研究了如何激发大型语言模型进行推理并生成答案时所依据的思维导图。研究发现，所生成的思维导图展现了大型语言模型基于知识本体的推理路径，从而为在生产环境中探查和评估大型语言模型推理提供了可能性。在三个问答数据集上的实验表明，思维导图提示方法带来了显著的性能提升。例如，使用思维导图提示GPT-3.5能持续超越GPT-4的表现。我们还证明，通过从知识图谱中检索结构化事实，思维导图方法能够优于一系列基于文档检索的提示方法，这得益于知识图谱提供的更准确、简洁和全面的知识。