Integrating new data into knowledge graphs (KG) typically involves different tasks that are executed within workflows or pipelines There are many possible pipelines for a specific integration problem but there is not yet a general approach to evaluate the overall quality and performance of such pipelines to be able to determine the best choices. We therefore propose a new benchmark KGI-Bench to evaluate integration pipelines that ingest different kinds of input data into an existing KG. We evaluate pipelines by analyzing their output, i.e., the updated KG, with the three complementary quality metrics coverage, correctness and consistency. We also provide benchmark datasets (seed KG, overlapping input data of three formats, reference KG as a ground truth) for the movie domain. To demonstrate the applicability and usefulness of the proposed benchmark, we comparatively evaluate 12 pipelines and analyze their behavior across different input data formats and design choices.

翻译：将新数据集成为知识图谱通常涉及构建工作流或管道中的不同任务。针对特定集成问题存在多种可能的流程，但目前尚无通用方法评估此类流程的整体质量与性能以确定最佳选择。为此，我们提出新基准KGI-Bench，用于评估将不同类型输入数据接入现有知识图谱的集成流程。我们通过分析流程输出（即更新后的知识图谱），采用覆盖度、正确性和一致性三个互补质量指标进行评估。同时提供面向电影领域的基准数据集（种子知识图谱、三种格式的重叠输入数据、作为标准答案的参考知识图谱）。为验证所提基准的适用性与实用性，我们对比评估了12种流程，并分析了它们在不同输入数据格式与设计选择下的行为特征。