Factual knowledge encoded in Pre-trained Language Models (PLMs) enriches their representations and justifies their use as knowledge bases. Previous work has focused on probing PLMs for factual knowledge by measuring how often they can correctly predict an object entity given a subject and a relation, and improving fact retrieval by optimizing the prompts used for querying PLMs. In this work, we consider a complementary aspect, namely the coherency of factual knowledge in PLMs, i.e., how often can PLMs predict the subject entity given its initial prediction of the object entity. This goes beyond evaluating how much PLMs know, and focuses on the internal state of knowledge inside them. Our results indicate that PLMs have low coherency using manually written, optimized and paraphrased prompts, but including an evidence paragraph leads to substantial improvement. This shows that PLMs fail to model inverse relations and need further enhancements to be able to handle retrieving facts from their parameters in a coherent manner, and to be considered as knowledge bases.

翻译：预训练语言模型（PLMs）中编码的事实知识丰富了其表征能力，并证明了其作为知识库的合理性。先前的研究侧重于通过测量模型在给定主体和关系时正确预测客体实体的频率来探查PLMs中的事实知识，并通过优化查询PLMs的提示来改进事实检索。本研究则关注一个互补的方面，即PLMs中事实知识的连贯性——当模型最初预测了客体实体后，其预测主体实体的频率如何。这超越了评估PLMs"知道多少"的范畴，转而聚焦于模型内部的知识状态。实验结果表明，无论使用人工编写、优化还是改写后的提示，PLMs的连贯性均较低，但加入证据段落可显著提升性能。这揭示了PLMs难以建模逆关系，需要进一步增强才能以连贯方式从其参数中检索事实，并被视为合格的知识库。