Knowledge-enhanced pre-trained language models (KEPLMs) leverage relation triples from knowledge graphs (KGs) and integrate these external data sources into language models via self-supervised learning. Previous works treat knowledge enhancement as two independent operations, i.e., knowledge injection and knowledge integration. In this paper, we propose to learn Knowledge-Enhanced language representations with Hierarchical Reinforcement Learning (KEHRL), which jointly addresses the problems of detecting positions for knowledge injection and integrating external knowledge into the model in order to avoid injecting inaccurate or irrelevant knowledge. Specifically, a high-level reinforcement learning (RL) agent utilizes both internal and prior knowledge to iteratively detect essential positions in texts for knowledge injection, which filters out less meaningful entities to avoid diverting the knowledge learning direction. Once the entity positions are selected, a relevant triple filtration module is triggered to perform low-level RL to dynamically refine the triples associated with polysemic entities through binary-valued actions. Experiments validate KEHRL's effectiveness in probing factual knowledge and enhancing the model's performance on various natural language understanding tasks.

翻译：知识增强的预训练语言模型（KEPLMs）利用来自知识图谱（KGs）的关系三元组，并通过自监督学习将这些外部数据源整合到语言模型中。先前的研究将知识增强视为两个独立的操作，即知识注入和知识集成。本文提出一种基于分层强化学习的知识增强语言表示学习方法（KEHRL），该方法联合处理知识注入位置检测和外部知识集成问题，以避免注入不准确或不相关的知识。具体而言，一个高层强化学习（RL）代理利用内部知识和先验知识，迭代地检测文本中用于知识注入的关键位置，从而过滤掉意义较弱的实体，避免偏离知识学习方向。一旦实体位置被选定，一个相关的三元组过滤模块将被触发，执行低层强化学习，通过二值化动作动态精炼与多义实体相关的三元组。实验验证了KEHRL在探测事实知识以及提升模型在多种自然语言理解任务性能方面的有效性。