Modern Large Language Models (LLMs) have shown impressive performances in user-facing tasks such as question answering, as well as consistent improvements in reasoning capabilities. Still, the way these models encode knowledge seems inherently flawed: by design, LLMs encode world-knowledge within their parameters. This way of representing knowledge is inherently opaque, difficult to debug and update, and prone to hallucinations. On the other hand, Knowledge Graphs can provide human-readable and easily editable world knowledge representations, and their application in knowledge-intensive tasks has consistently proven beneficial to downstream performance. Nonetheless, current integration techniques require extensive retraining or finetuning. To overcome this issue, we introduce KoRe, a methodology to encode 1-hop sub-graphs into compact discrete knowledge tokens and inject them into a LLM backbone. We test the proposed approach on three established benchmarks, and report competitive performances coupled with a significant reduction (up to 10x) in token usage. Our results show that compact discrete KG representations can efficiently and effectively be used to ground modern LLMs.

翻译：现代大型语言模型（LLM）在问答等面向用户的任务中展现了令人瞩目的性能，并在推理能力上持续取得改进。然而，这些模型编码知识的方式存在固有缺陷：LLM 本质上将世界知识编码在其参数中。这种知识表示方式天生不透明、难以调试和更新，且容易产生幻觉。另一方面，知识图谱能够提供人类可读且易于编辑的世界知识表示，在知识密集型任务中始终证实有助于提升下游性能。尽管如此，当前的集成技术仍需大量重训练或微调。为解决此问题，我们提出 KoRe——一种将单跳子图编码为紧凑离散知识令牌，并将其注入 LLM 主干网络的方法。我们在三个公认基准上测试了所提方法，并报告了具有竞争力的性能，同时令牌使用量显著降低（最高达 10 倍）。我们的结果表明，紧凑的离散知识图谱表示可高效且有效地用于引导现代 LLM。