To address the issue of insufficient knowledge and the tendency to generate hallucination in Large Language Models (LLMs), numerous studies have endeavored to integrate LLMs with Knowledge Graphs (KGs). However, all these methods are evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where the factual triples involved in each question are entirely covered by the given KG. In this situation, LLM mainly acts as an agent to find answer entities by exploring the KG, rather than effectively integrating internal and external knowledge sources. However, in real-world scenarios, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, in this paper, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the given KG doesn't include all the factual triples involved in each question. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG) that can generate new factual triples while exploring on KGs. Specifically, we propose a selecting-generating-answering framework, which not only treat the LLM as an agent to explore on KGs, but also treat it as a KG to generate new facts based on the explored subgraph and its inherent knowledge. Experimental results on two datasets demonstrate that our GoG can solve IKGQA to a certain extent, while almost all previous methods cannot perform well on IKGQA.

翻译：为解决大语言模型（LLMs）知识不足及易产生幻觉的问题，大量研究致力于将LLMs与知识图谱（KGs）进行融合。然而，这些方法均是在传统知识图谱问答（KGQA）场景下评估的，此类场景中的KG是完整的，每个问题所涉及的事实三元组均能被给定KG完整覆盖。在此情况下，LLM主要充当智能体通过探索KG来寻找答案实体，而非有效融合内部知识与外部知识源。但在现实场景中，KG往往不完整，无法覆盖回答问题所需的全部知识。为模拟真实场景并评估LLM整合内外部知识的能力，本文提出在不完全知识图谱（IKGQA）下利用LLM进行问答，其中给定KG不包含每个问题涉及的全部事实三元组。为处理IKGQA，我们提出一种免训练方法——图上生成（Generate-on-Graph, GoG），该方法可在探索KG的同时生成新的事实三元组。具体而言，我们提出"选择-生成-回答"框架，该框架不仅将LLM作为探索KG的智能体，还将其作为知识图谱，基于探索到的子图及其固有知识生成新事实。两个数据集上的实验结果表明，我们的GoG能够在一定程度上解决IKGQA问题，而几乎所有现有方法在此任务上均表现不佳。