Large language models (LLMs), such as GPT3.5, GPT4 and LLAMA2 perform surprisingly well and outperform human experts on many tasks. However, in many domain-specific evaluations, these LLMs often suffer from hallucination problems due to insufficient training of relevant corpus. Furthermore, fine-tuning large models may face problems such as the LLMs are not open source or the construction of high-quality domain instruction is difficult. Therefore, structured knowledge databases such as knowledge graph can better provide domain back- ground knowledge for LLMs and make full use of the reasoning and analysis capabilities of LLMs. In some previous works, LLM was called multiple times to determine whether the current triplet was suitable for inclusion in the subgraph when retrieving subgraphs through a question. Especially for the question that require a multi-hop reasoning path, frequent calls to LLM will consume a lot of computing power. Moreover, when choosing the reasoning path, LLM will be called once for each step, and if one of the steps is selected incorrectly, it will lead to the accumulation of errors in the following steps. In this paper, we integrated and optimized a pipeline for selecting reasoning paths from KG based on LLM, which can reduce the dependency on LLM. In addition, we propose a simple and effective subgraph retrieval method based on chain of thought (CoT) and page rank which can returns the paths most likely to contain the answer. We conduct experiments on three datasets: GenMedGPT-5k [14], WebQuestions [2], and CMCQA [21]. Finally, RoK can demonstrate that using fewer LLM calls can achieve the same results as previous SOTAs models.

翻译：大型语言模型（LLMs），例如GPT3.5、GPT4和LLAMA2，在许多任务上表现出惊人的性能，甚至超越人类专家。然而，在许多特定领域的评估中，由于相关语料训练不足，这些LLMs常出现幻觉问题。此外，微调大模型可能面临诸如LLM未开源或高质量领域指令构建困难等问题。因此，知识图谱等结构化知识数据库能更好地为LLMs提供领域背景知识，并充分利用LLMs的推理与分析能力。在以往的一些工作中，当通过问题检索子图时，需要多次调用LLM来判断当前三元组是否适合纳入子图。特别是对于需要多跳推理路径的问题，频繁调用LLM会消耗大量计算资源。此外，在选择推理路径时，每一步都需要调用一次LLM，若某一步选择错误，将导致后续步骤的错误累积。本文中，我们集成并优化了一个基于LLM从知识图谱中选择推理路径的流水线，以减少对LLM的依赖。此外，我们提出了一种基于思维链（CoT）和PageRank的简单有效的子图检索方法，该方法能返回最可能包含答案的路径。我们在三个数据集上进行了实验：GenMedGPT-5k [14]、WebQuestions [2]和CMCQA [21]。最后，RoK证明了使用更少的LLM调用即可达到与先前SOTA模型相同的结果。