A characteristic feature of human semantic cognition is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties (can breathe) from superordinate concepts (animal) to their subordinates (dog) -- i.e. demonstrate property inheritance. In this paper, we present COMPS, a collection of minimal pair sentences that jointly tests pre-trained language models (PLMs) on their ability to attribute properties to concepts and their ability to demonstrate property inheritance behavior. Analyses of 22 different PLMs on COMPS reveal that they can easily distinguish between concepts on the basis of a property when they are trivially different, but find it relatively difficult when concepts are related on the basis of nuanced knowledge representations. Furthermore, we find that PLMs can demonstrate behavior consistent with property inheritance to a great extent, but fail in the presence of distracting information, which decreases the performance of many models, sometimes even below chance. This lack of robustness in demonstrating simple reasoning raises important questions about PLMs' capacity to make correct inferences even when they appear to possess the prerequisite knowledge.

翻译：人类语义认知的一个典型特征在于，不仅能存储和检索经验中观察到的概念属性，还能促进属性（如“能呼吸”）从上位概念（如“动物”）向下位概念（如“狗”）的继承——即展示属性继承行为。本文提出COMPS——一组极小对句子集合，用于联合测试预训练语言模型（PLMs）对概念进行属性归因以及展现属性继承行为的能力。对22种不同PLMs在COMPS上的分析表明：当概念与属性存在显著差异时，模型能轻易区分；但当概念基于细微知识表征相关联时，模型则相对困难。此外，我们发现PLMs能在很大程度上表现出与属性继承一致的行为，但在存在干扰信息时失败，导致许多模型性能下降，甚至低于随机水平。这种展现简单推理时鲁棒性的缺失，对PLMs即便看似具备先决知识时能否做出正确推理提出了重要质疑。