To address the issues of insufficient knowledge and hallucination in Large Language Models (LLMs), numerous studies have explored integrating LLMs with Knowledge Graphs (KGs). However, these methods are typically evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where all factual triples required for each question are entirely covered by the given KG. In such cases, LLMs primarily act as an agent to find answer entities within the KG, rather than effectively integrating the internal knowledge of LLMs and external knowledge sources such as KGs. In fact, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate these real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the provided KG lacks some of the factual triples for each question, and construct corresponding datasets. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG), which can generate new factual triples while exploring KGs. Specifically, GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA. Experimental results on two datasets demonstrate that our GoG outperforms all previous methods.

翻译：为解决大型语言模型（LLM）中知识不足与幻觉生成的问题，大量研究探索了将LLM与知识图谱（KG）相结合的方法。然而，这些方法通常在基于完整知识图谱的传统知识图谱问答（KGQA）场景中进行评估，其中每个问题所需的所有事实三元组均被给定KG完全覆盖。在此类情况下，LLM主要充当在KG中寻找答案实体的智能体，而非有效整合LLM内部知识与KG等外部知识源。实际上，KG往往无法完整覆盖回答问题所需的全部知识。为模拟此类现实场景并评估LLM整合内外知识的能力，我们提出在不完整知识图谱（IKGQA）条件下利用LLM进行问答，其中提供的KG对每个问题缺失部分事实三元组，并构建了相应数据集。针对IKGQA任务，我们提出一种无需训练的方法——基于图的生成（GoG），该方法能在探索KG的同时生成新的事实三元组。具体而言，GoG通过“思考-搜索-生成”框架进行推理，将LLM同时视为IKGQA中的智能体与知识图谱。在两个数据集上的实验结果表明，我们的GoG方法优于所有现有方法。