Several recent efforts have been devoted to enhancing pre-trained language models (PLMs) by utilizing extra heterogeneous knowledge in knowledge graphs (KGs) and achieved consistent improvements on various knowledge-driven NLP tasks. However, most of these knowledge-enhanced PLMs embed static sub-graphs of KGs ("knowledge context"), regardless of that the knowledge required by PLMs may change dynamically according to specific text ("textual context"). In this paper, we propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context for PLMs, which can avoid the effect of redundant and ambiguous knowledge in KGs that cannot match the input text. Our experimental results show that Coke outperforms various baselines on typical knowledge-driven NLP tasks, indicating the effectiveness of utilizing dynamic knowledge context for language understanding. Besides the performance improvements, the dynamically selected knowledge in Coke can describe the semantics of text-related knowledge in a more interpretable form than the conventional PLMs. Our source code and datasets will be available to provide more details for Coke.

翻译：近年来，多项研究致力于利用知识图谱（KG）中的额外异构知识增强预训练语言模型（PLM），并在各类知识驱动的自然语言处理任务中取得持续改进。然而，现有知识增强型PLM大多直接嵌入知识图谱的静态子图（"知识上下文"），忽视了PLM所需知识可能随具体文本（"文本上下文"）动态变化的事实。本文提出名为Coke的新型框架，能够根据PLM的文本上下文动态选择上下文知识并嵌入知识上下文，从而避免知识图谱中与输入文本不匹配的冗余和歧义知识带来的负面影响。实验结果表明，Coke在典型知识驱动的NLP任务上优于多种基线模型，验证了利用动态知识上下文进行语言理解的有效性。除性能提升外，Coke动态选择的知识能以比传统PLM更具可解释性的形式描述文本相关知识的语义。我们将开放源代码和数据集，以提供Coke的更多实现细节。