Despite the significant progress made by transformer models in machine reading comprehension tasks, they still fall short in handling complex reasoning tasks due to the absence of explicit knowledge in the input sequence. To address this limitation, many recent works have proposed injecting external knowledge into the model. However, selecting relevant external knowledge, ensuring its availability, and requiring additional processing steps remain challenging. In this paper, we introduce a novel attention pattern that integrates reasoning knowledge derived from a heterogeneous graph into the transformer architecture without relying on external knowledge. The proposed attention pattern comprises three key elements: global-local attention for word tokens, graph attention for entity tokens that exhibit strong attention towards tokens connected in the graph as opposed to those unconnected, and the consideration of the type of relationship between each entity token and word token. This results in optimized attention between the two if a relationship exists. The pattern is coupled with special relative position labels, allowing it to integrate with LUKE's entity-aware self-attention mechanism. The experimental findings corroborate that our model outperforms both the cutting-edge LUKE-Graph and the baseline LUKE model on the ReCoRD dataset that focuses on commonsense reasoning.

翻译：尽管Transformer模型在机器阅读理解任务中取得了显著进展，但由于输入序列中缺乏显式知识，它们仍难以处理复杂推理任务。为突破这一局限，近期许多研究提出将外部知识注入模型。然而，筛选相关外部知识、确保其可获得性以及额外处理步骤的引入仍面临挑战。本文提出一种新型注意力模式，该模式将异构图中推导出的推理知识融入Transformer架构，无需依赖外部知识。所提出的注意力模式包含三个关键要素：针对词元的全局-局部注意力、对图结构中连接词元表现出强注意力（相较于未连接词元）的实体词元图注意力，以及每个实体词元与词元之间关系类型的考量——若存在关系则优化两者间的注意力。该模式结合特殊相对位置标签，可集成至LUKE的实体感知自注意力机制。实验结果证实，在聚焦常识推理的ReCoRD数据集上，我们的模型优于前沿的LUKE-Graph模型及基线LUKE模型。