Pre-trained Language Models (PLMs) which are trained on large text corpus via self-supervised learning method, have yielded promising performance on various tasks in Natural Language Processing (NLP). However, though PLMs with huge parameters can effectively possess rich knowledge learned from massive training text and benefit downstream tasks at the fine-tuning stage, they still have some limitations such as poor reasoning ability due to the lack of external knowledge. Research has been dedicated to incorporating knowledge into PLMs to tackle these issues. In this paper, we present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs) to provide a clear insight into this thriving field. We introduce appropriate taxonomies respectively for Natural Language Understanding (NLU) and Natural Language Generation (NLG) to highlight these two main tasks of NLP. For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG), and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods. Finally, we point out some promising future directions of KE-PLMs.

翻译：预训练语言模型（PLMs）通过在大规模文本语料上采用自监督学习方法进行训练，已在自然语言处理（NLP）的各类任务中展现出优异性能。然而，尽管拥有海量参数的PLMs能够从训练文本中有效获取丰富知识，并在微调阶段为下游任务带来增益，但其仍存在推理能力不足等局限性——这源于外部知识的缺乏。近年来，研究者致力于将知识融入PLMs以应对上述问题。本文系统综述了知识增强型预训练语言模型（KE-PLMs）领域的研究进展，旨在为这一蓬勃发展的研究方向提供清晰的学术洞察。我们针对自然语言理解（NLU）与自然语言生成（NLG）两大NLP核心任务，分别构建了合理的分类体系：在NLU维度，将知识类型划分为语言知识、文本知识、知识图谱（KG）与规则知识四类；对于NLG任务，则提出基于知识图谱与基于检索的两种KE-PLM方法分类。最后，我们指出了KE-PLMs领域若干具有前景的未来研究方向。