Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system (edge et al., 2024). Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher. We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics.

翻译：从图数据中检索对于增强大语言模型（LLM）的开放领域知识和私有企业数据至关重要，同时也是近期GraphRAG系统（edge等人，2024）的核心组件。尽管知识图谱和知识库问答研究已开展数十年，主流LLM框架（如Langchain和LlamaIndex）对现代百科全书式知识图谱（如Wikidata）的检索支持仍极为有限。本文通过分析指出，现代RDF知识图谱（如Wikidata、Freebase）因模式规模远超典型LLM上下文窗口、使用资源标识符、关系类型重叠及缺乏规范化等问题，导致其LLM检索效率低下。为此，我们提出在底层RDF图谱之上构建属性图视图，使LLM能够通过Cypher查询语言实现高效查询。基于Wikidata，我们实现了该构想并推出CypherBench——首个包含11个大规模跨领域属性图（含780万个实体）及超10,000道问题的基准测试集。为实现这一目标，我们攻克了多项关键技术挑战，包括开发RDF到属性图的转换引擎、构建文本到Cypher任务的系统化生成流程，以及设计新型评估指标。