This study explores the use of Large Language Models (LLMs) for automatic evaluation of knowledge graph (KG) completion models. Historically, validating information in KGs has been a challenging task, requiring large-scale human annotation at prohibitive cost. With the emergence of general-purpose generative AI and LLMs, it is now plausible that human-in-the-loop validation could be replaced by a generative agent. We introduce a framework for consistency and validation when using generative models to validate knowledge graphs. Our framework is based upon recent open-source developments for structural and semantic validation of LLM outputs, and upon flexible approaches to fact checking and verification, supported by the capacity to reference external knowledge sources of any kind. The design is easy to adapt and extend, and can be used to verify any kind of graph-structured data through a combination of model-intrinsic knowledge, user-supplied context, and agents capable of external knowledge retrieval.

翻译：本研究探索利用大型语言模型(LLMs)自动评估知识图谱(KG)补全模型。历史上，验证知识图谱中的信息一直是一项具有挑战性的任务，需要大规模人工标注且成本高昂。随着通用生成式人工智能和大型语言模型的出现，用生成式智能体取代人在回路中的验证方式已具有可行性。我们提出了一套在使用生成式模型验证知识图谱时保障一致性与准确性的框架。该框架基于近期开源技术对LLM输出进行结构与语义验证，并采用灵活的事实核查与验证方法，同时具备引用任意外部知识源的能力。其设计易于适配和扩展，可通过模型固有知识、用户提供的上下文以及具备外部知识检索能力的智能体组合，验证任意类型的图结构数据。