This paper presents an exhaustive quantitative and qualitative evaluation of Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning. We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference. Empirically, our findings suggest that LLMs, represented by GPT-4, are more suited as inference assistants rather than few-shot information extractors. Specifically, while GPT-4 exhibits good performance in tasks related to KG construction, it excels further in reasoning tasks, surpassing fine-tuned models in certain cases. Moreover, our investigation extends to the potential generalization ability of LLMs for information extraction, leading to the proposition of a Virtual Knowledge Extraction task and the development of the corresponding VINE dataset. Based on these empirical findings, we further propose AutoKG, a multi-agent-based approach employing LLMs and external sources for KG construction and reasoning. We anticipate that this research can provide invaluable insights for future undertakings in the field of knowledge graphs. The code and datasets are in https://github.com/zjunlp/AutoKG.

翻译：本文对大型语言模型（LLMs）在知识图谱（KG）构建与推理中的应用进行了详尽的定量与定性评估。我们围绕实体与关系抽取、事件抽取、链接预测和问答四项代表性任务，在八个不同数据集上开展实验，全面探究了LLMs在知识图谱构建与推理领域中的表现。实验结果表明，以GPT-4为代表的LLMs更适合作为推理辅助工具，而非少样本信息抽取器。具体而言，GPT-4在KG构建相关任务中表现良好，而在推理任务中表现更为优异，在某些情况下甚至优于微调模型。此外，我们进一步研究了LLMs在信息抽取中的潜在泛化能力，提出了虚拟知识抽取任务并构建了相应的VINE数据集。基于这些实证发现，我们进一步提出AutoKG——一种基于多智能体的方法，利用LLMs和外部资源进行KG构建与推理。我们期待本研究能为未来知识图谱领域的工作提供宝贵见解。相关代码与数据集详见https://github.com/zjunlp/AutoKG。