Pre-trained language models (PLMs) have been prevailing in state-of-the-art methods for natural language processing, and knowledge-enhanced PLMs are further proposed to promote model performance in knowledge-intensive tasks. However, conceptual knowledge, one essential kind of knowledge for human cognition, still remains understudied in this line of research. This limits PLMs' performance in scenarios requiring human-like cognition, such as understanding long-tail entities with concepts. In this paper, we propose ConcEPT, which stands for Concept-Enhanced Pre-Training for language models, to infuse conceptual knowledge into PLMs. ConcEPT exploits external taxonomies with entity concept prediction, a novel pre-training objective to predict the concepts of entities mentioned in the pre-training contexts. Unlike previous concept-enhanced methods, ConcEPT can be readily adapted to various downstream applications without entity linking or concept mapping. Results of extensive experiments show the effectiveness of ConcEPT in four tasks such as entity typing, which validates that our model gains improved conceptual knowledge with concept-enhanced pre-training.

翻译：摘要：预训练语言模型（PLMs）在自然语言处理的最先进方法中占据主导地位，为进一步提升知识密集型任务中的模型性能，知识增强型PLMs被相继提出。然而，概念知识作为人类认知的一种基本知识类型，在该研究方向中仍缺乏充分研究。这限制了PLMs在需要类似人类认知的场景中的表现，例如理解涉及概念的长尾实体。本文提出ConcEPT（概念增强语言模型预训练），旨在将概念知识注入PLMs。ConcEPT利用外部分类体系，通过实体概念预测这一新颖的预训练目标，预测预训练文本中提及实体的概念。与以往概念增强方法不同，ConcEPT无需实体链接或概念映射即可直接适配各类下游应用。大量实验结果表明，ConcEPT在实体分类等四项任务中表现优异，验证了通过概念增强预训练，模型能够获得更完善的概念知识。