LLMs usually exhibit limitations in their ability to incorporate new knowledge, the generation of hallucinations, and the transparency of their decision-making process. In this paper, we explore how to prompt LLMs with knowledge graphs (KG), working as a remedy to engage LLMs with up-to-date knowledge and elicit the reasoning pathways from LLMs. Specifically, we build a prompting pipeline that endows LLMs with the capability of comprehending KG inputs and inferring with a combined implicit knowledge and the retrieved external knowledge. In addition, we investigate eliciting the mind map on which LLMs perform the reasoning and generate the answers. It is identified that the produced mind map exhibits the reasoning pathways of LLMs grounded on the ontology of knowledge, hence bringing the prospects of probing and gauging LLM inference in production. The experiments on three question & answering datasets also show that MindMap prompting leads to a striking empirical gain. For instance, prompting a GPT-3.5 with MindMap yields an overwhelming performance over GPT-4 consistently. We also demonstrate that with structured facts retrieved from KG, MindMap can outperform a series of prompting-with-document-retrieval methods, benefiting from more accurate, concise, and comprehensive knowledge from KGs.

翻译：大型语言模型（LLMs）通常在融入新知识的能力、生成幻觉以及决策过程的透明度方面存在局限。本文探讨如何利用知识图谱（KG）提示LLMs，以弥补这些不足，使LLMs能够结合最新知识并激发其推理路径。具体而言，我们构建了一个提示流水线，赋予LLMs理解知识图谱输入的能力，并使其能够结合隐含知识与检索到的外部知识进行推理。此外，我们研究了如何激发LLMs用于推理和生成答案的思维图谱。实验发现，生成的思维图谱展示了LLMs基于知识本体进行推理的路径，从而为在生产环境中探测和衡量LLM推理提供了前景。在三个问答数据集上的实验表明，MindMap提示方法带来了显著的性能提升。例如，使用MindMap提示GPT-3.5可稳定地超越GPT-4的表现。我们还证明，通过从知识图谱中检索结构化事实，MindMap能够优于一系列基于文档检索的提示方法，这得益于知识图谱提供的更准确、简洁且全面的知识。