A backbone of knowledge graphs are their class membership relations, which assign entities to a given class. As part of the knowledge engineering process, we propose a new method for evaluating the quality of these relations by processing descriptions of a given entity and class using a zero-shot chain-of-thought classifier that uses a natural language intensional definition of a class. We evaluate the method using two publicly available knowledge graphs, Wikidata and CaLiGraph, and 7 large language models. Using the gpt-4-0125-preview large language model, the method's classification performance achieves a macro-averaged F1-score of 0.830 on data from Wikidata and 0.893 on data from CaLiGraph. Moreover, a manual analysis of the classification errors shows that 40.9% of errors were due to the knowledge graphs, with 16.0% due to missing relations and 24.9% due to incorrectly asserted relations. These results show how large language models can assist knowledge engineers in the process of knowledge graph refinement. The code and data are available on Github.

翻译：知识图谱的核心支撑是类成员关系，该关系将实体分配至特定类别。作为知识工程流程的组成部分，我们提出一种评估此类关系质量的新方法：通过处理给定实体与类别的描述信息，利用基于自然语言内涵定义的零样本思维链分类器进行分析。我们采用维基数据（Wikidata）和CaLiGraph两个公开知识图谱，并结合7种大语言模型进行方法验证。采用gpt-4-0125-preview大语言模型时，该方法在维基数据上实现宏平均F1分数0.830，在CaLiGraph数据上达到0.893。分类误差的微观分析表明，40.9%的误差源于知识图谱本身（其中16.0%为缺失关系，24.9%为错误断言关系）。这些结果揭示了大语言模型在知识图谱精炼过程中辅助知识工程师的潜力。相关代码与数据已发布于GitHub。