The paradigm of Transformers using the self-attention mechanism has manifested its advantage in learning graph-structured data. Yet, Graph Transformers are capable of modeling full range dependencies but are often deficient in extracting information from locality. A common practice is to utilize Message Passing Neural Networks (MPNNs) as an auxiliary to capture local information, which however are still inadequate for comprehending substructures. In this paper, we present a purely attention-based architecture, namely Focal and Full-Range Graph Transformer (FFGT), which can mitigate the loss of local information in learning global correlations. The core component of FFGT is a new mechanism of compound attention, which combines the conventional full-range attention with K-hop focal attention on ego-nets to aggregate both global and local information. Beyond the scope of canonical Transformers, the FFGT has the merit of being more substructure-aware. Our approach enhances the performance of existing Graph Transformers on various open datasets, while achieves compatible SOTA performance on several Long-Range Graph Benchmark (LRGB) datasets even with a vanilla transformer. We further examine influential factors on the optimal focal length of attention via introducing a novel synthetic dataset based on SBM-PATTERN.

翻译：采用自注意力机制的变换器范式在学习图结构数据方面展现出优势。然而，图变换器虽能建模全局依赖关系，却常缺乏从局部区域提取信息的能力。常见做法是利用消息传递神经网络作为辅助手段来捕获局部信息，但这仍不足以充分理解子结构。本文提出一种纯注意力架构——焦点与全范围图变换器（FFGT），该架构能在学习全局关联时缓解局部信息损失问题。FFGT的核心组件是一种新型复合注意力机制，它将传统全范围注意力与基于自我网络的K跳焦点注意力相结合，从而同时聚合全局与局部信息。超越标准变换器的范畴，FFGT在子结构感知方面具有优势。我们的方法在多个开放数据集上提升了现有图变换器的性能，即使采用基础变换器架构，也能在多个长程图基准（LRGB）数据集上达到可比的当前最优性能。此外，我们通过引入基于SBM-PATTERN的新型合成数据集，进一步探究了影响注意力最佳焦点长度的关键因素。