With the recent spike in the number and availability of Large Language Models (LLMs), it has become increasingly important to provide large and realistic benchmarks for evaluating Knowledge Graph Question Answering (KBQA) systems. So far the majority of benchmarks rely on pattern-based SPARQL query generation approaches. The subsequent natural language (NL) question generation is conducted through crowdsourcing or other automated methods, such as rule-based paraphrasing or NL question templates. Although some of these datasets are of considerable size, their pitfall lies in their pattern-based generation approaches, which do not always generalize well to the vague and linguistically diverse questions asked by humans in real-world contexts. In this paper, we introduce Spider4SPARQL - a new SPARQL benchmark dataset featuring 9,693 previously existing manually generated NL questions and 4,721 unique, novel, and complex SPARQL queries of varying complexity. In addition to the NL/SPARQL pairs, we also provide their corresponding 166 knowledge graphs and ontologies, which cover 138 different domains. Our complex benchmark enables novel ways of evaluating the strengths and weaknesses of modern KGQA systems. We evaluate the system with state-of-the-art KGQA systems as well as LLMs, which achieve only up to 45\% execution accuracy, demonstrating that Spider4SPARQL is a challenging benchmark for future research.

翻译：随着大语言模型（LLM）的数量和可用性近期激增，提供大规模且逼真的基准来评估知识图谱问答（KGQA）系统已变得日益重要。迄今为止，大多数基准依赖于基于模式的SPARQL查询生成方法。随后的自然语言（NL）问题生成通过众包或其他自动化方法（如基于规则的释义或NL问题模板）完成。尽管其中部分数据集规模可观，但它们的缺陷在于基于模式的生成方法，该方法难以良好泛化至人类在真实场景中提出的模糊且语言多样的问题。本文介绍了Spider4SPARQL——一个新的SPARQL基准数据集，包含9,693个先前人工生成的NL问题以及4,721个独特、新颖且复杂度各异的SPARQL查询。除NL/SPARQL配对外，我们还提供了对应的166个知识图谱和本体，覆盖138个不同领域。我们这一复杂基准为评估现代KGQA系统的优劣势提供了新途径。我们采用最先进的KGQA系统及LLM对该系统进行评估，其执行准确率最高仅为45%，表明Spider4SPARQL是未来研究中具有挑战性的基准。