Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. We refer to pre-trained language models with knowledge injection as knowledge-enhanced pre-trained language models (KEPLMs). These models demonstrate deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPLMs in NLP. We first discuss the advancements in pre-trained language models and knowledge representation learning. Then we systematically categorize existing KEPLMs from three different perspectives. Finally, we outline some potential directions of KEPLMs for future research.

翻译：预训练语言模型通过自监督学习在大规模文本语料上学习信息丰富的词表示，在自然语言处理领域微调后取得了显著性能。然而，这些模型存在鲁棒性差和缺乏可解释性的问题。我们将经过知识注入的预训练语言模型称为知识增强预训练语言模型（KEPLMs）。这类模型展现出深层理解与逻辑推理能力，并引入了可解释性。本综述对自然语言处理中的KEPLMs进行了全面概述。首先讨论预训练语言模型与知识表示学习的进展，随后从三个不同视角对现有KEPLMs进行系统分类，最后展望KEPLMs未来的研究方向。