The ability to summarize and organize knowledge into abstract concepts is key to learning and reasoning. Many industrial applications rely on the consistent and systematic use of concepts, especially when dealing with decision-critical knowledge. However, we demonstrate that, when methodically questioned, large language models (LLMs) often display and demonstrate significant inconsistencies in their knowledge. Computationally, the basic aspects of the conceptualization of a given domain can be represented as Is-A hierarchies in a knowledge graph (KG) or ontology, together with a few properties or axioms that enable straightforward reasoning. We show that even simple ontologies can be used to reveal conceptual inconsistencies across several LLMs. We also propose strategies that domain experts can use to evaluate and improve the coverage of key domain concepts in LLMs of various sizes. In particular, we have been able to significantly enhance the performance of LLMs of various sizes with openly available weights using simple knowledge-graph (KG) based prompting strategies.

翻译：将知识总结并组织为抽象概念的能力是学习和推理的关键。许多工业应用依赖于概念的一致性和系统性使用，尤其是在处理决策关键知识时。然而，我们发现，当被系统性质询时，大型语言模型（LLMs）在其知识方面经常表现出显著的不一致性。从计算角度，给定领域概念化的基本方面可以表示为知识图谱（KG）或本体中的Is-A层次结构，辅以一些支持直接推理的属性或公理。我们证明，即使是简单的本体也能用于揭示多个LLMs中的概念不一致性。我们还提出了领域专家可用于评估和改进不同规模LLMs中关键领域概念覆盖度的策略。特别是，通过使用基于知识图谱（KG）的简单提示策略，我们显著提升了多个具有公开权重的不同规模LLMs的性能。