The conventional process of building Ontologies and Knowledge Graphs (KGs) heavily relies on human domain experts to define entities and relationship types, establish hierarchies, maintain relevance to the domain, fill the ABox (or populate with instances), and ensure data quality (including amongst others accuracy and completeness). On the other hand, Large Language Models (LLMs) have recently gained popularity for their ability to understand and generate human-like natural language, offering promising ways to automate aspects of this process. This work explores the (semi-)automatic construction of KGs facilitated by open-source LLMs. Our pipeline involves formulating competency questions (CQs), developing an ontology (TBox) based on these CQs, constructing KGs using the developed ontology, and evaluating the resultant KG with minimal to no involvement of human experts. We showcase the feasibility of our semi-automated pipeline by creating a KG on deep learning methodologies by exploiting scholarly publications. To evaluate the answers generated via Retrieval-Augmented-Generation (RAG) as well as the KG concepts automatically extracted using LLMs, we design a judge LLM, which rates the generated content based on ground truth. Our findings suggest that employing LLMs could potentially reduce the human effort involved in the construction of KGs, although a human-in-the-loop approach is recommended to evaluate automatically generated KGs.

翻译：传统构建本体和知识图谱的流程高度依赖人类领域专家，用以定义实体和关系类型、建立层次结构、保持领域相关性、填充ABox（或实例化），以及确保数据质量（包括准确性和完整性等）。另一方面，大型语言模型因其理解和生成类人自然语言的能力近年来广受欢迎，为自动化该过程的某些环节提供了有前景的途径。本研究探索了由开源LLM辅助的（半）自动知识图谱构建过程。我们的流水线包括：制定能力问题、基于这些CQ开发本体、利用所开发本体构建知识图谱，并在最少或无需人类专家参与的情况下评估生成的KG。我们通过利用学术出版物构建关于深度学习方法的KG，展示了半自动化流水线的可行性。为评估通过检索增强生成生成的答案以及由LLM自动提取的KG概念，我们设计了一个裁判LLM，该模型根据基准真实情况对生成内容进行评分。研究结果表明，采用LLM可能减少构建KG所需的人力投入，但仍建议采用人机协同方法来评估自动生成的KG。