Extrapolation in Large language models (LLMs) for open-ended inquiry encounters two pivotal issues: (1) hallucination and (2) expensive training costs. These issues present challenges for LLMs in specialized domains and personalized data, requiring truthful responses and low fine-tuning costs. Existing works attempt to tackle the problem by augmenting the input of a smaller language model with information from a knowledge graph (KG). However, they have two limitations: (1) failing to extract relevant information from a large one-hop neighborhood in KG and (2) applying the same augmentation strategy for KGs with different characteristics that may result in low performance. Moreover, open-ended inquiry typically yields multiple responses, further complicating extrapolation. We propose a new task, the extreme multi-label KG link prediction task, to enable a model to perform extrapolation with multiple responses using structured real-world knowledge. Our retriever identifies relevant one-hop neighbors by considering entity, relation, and textual data together. Our experiments demonstrate that (1) KGs with different characteristics require different augmenting strategies, and (2) augmenting the language model's input with textual data improves task performance significantly. By incorporating the retrieval-augmented framework with KG, our framework, with a small parameter size, is able to extrapolate based on a given KG. The code can be obtained on GitHub: https://github.com/exiled1143/Retrieval-Augmented-Language-Model-for-Multi-Label-Knowledge-Graph-Link-Prediction.git

翻译：大语言模型在开放式查询中实现外推面临两个关键问题：（1）幻觉现象和（2）高昂的训练成本。这些问题使得大语言模型在专业领域和个性化数据应用中面临挑战，需要同时满足真实响应和低成本微调的要求。现有工作尝试通过将知识图谱信息增强至较小语言模型的输入来解决问题，但存在两个局限性：（1）未能从知识图谱的大规模一跳邻域中提取相关信息；（2）对不同特征的知识图谱采用相同的增强策略，可能导致性能低下。此外，开放式查询通常会产生多个响应，进一步增加了外推的复杂性。我们提出了一项新任务——极端多标签知识图谱链接预测任务，使模型能够利用结构化现实世界知识进行多响应外推。我们的检索器通过联合考量实体、关系和文本数据来识别相关的一跳邻域。实验证明：（1）不同特征的知识图谱需要不同的增强策略；（2）在语言模型输入中增加文本数据可显著提升任务性能。通过将检索增强框架与知识图谱相结合，我们的框架在较小参数规模下能够根据给定知识图谱进行外推。代码可在GitHub获取：https://github.com/exiled1143/Retrieval-Augmented-Language-Model-for-Multi-Label-Knowledge-Graph-Link-Prediction.git