Incorporating prior knowledge can improve existing pre-training models in cloze-style machine reading and has become a new trend in recent studies. Notably, most of the existing models have integrated external knowledge graphs (KG) and transformer-based models, such as BERT into a unified data structure. However, selecting the most relevant ambiguous entities in KG and extracting the best subgraph remains a challenge. In this paper, we propose the LUKE-Graph, a model that builds a heterogeneous graph based on the intuitive relationships between entities in a document without using any external KG. We then use a Relational Graph Attention (RGAT) network to fuse the graph's reasoning information and the contextual representation encoded by the pre-trained LUKE model. In this way, we can take advantage of LUKE, to derive an entity-aware representation; and a graph model - to exploit relation-aware representation. Moreover, we propose Gated-RGAT by augmenting RGAT with a gating mechanism that regulates the question information for the graph convolution operation. This is very similar to human reasoning processing because they always choose the best entity candidate based on the question information. Experimental results demonstrate that the LUKE-Graph achieves state-of-the-art performance on the ReCoRD dataset with commonsense reasoning.

翻译：融入先验知识可改进现有预训练模型在完形填空式机器阅读中的表现，并已成为近年研究的新趋势。值得注意的是，现有大部分模型已将外部知识图谱与基于Transformer的模型（如BERT）整合为统一数据结构。然而，如何选择知识图谱中最相关的歧义实体并提取最优子图仍具挑战性。本文提出LUKE-Graph模型，该模型基于文档中实体间的直觉关系构建异构图，无需使用任何外部知识图谱。随后，我们采用关系图注意力网络融合图的推理信息与预训练LUKE模型编码的上下文表示。通过这种方式，我们既能利用LUKE获取实体感知表示，又能借助图模型发掘关系感知表示。此外，我们通过向RGAT引入门控机制来调节图卷积操作的问题信息，由此提出门控RGAT。该机制与人类推理过程高度相似——人类总是基于问题信息选择最佳候选实体。实验结果表明，LUKE-Graph在需要常识推理的ReCoRD数据集上取得了最先进性能。