Learning representations on large-sized graphs is a long-standing challenge due to the inter-dependence nature involved in massive data points. Transformers, as an emerging class of foundation encoders for graph-structured data, have shown promising performance on small graphs due to its global attention capable of capturing all-pair influence beyond neighboring nodes. Even so, existing approaches tend to inherit the spirit of Transformers in language and vision tasks, and embrace complicated models by stacking deep multi-head attentions. In this paper, we critically demonstrate that even using a one-layer attention can bring up surprisingly competitive performance across node property prediction benchmarks where node numbers range from thousand-level to billion-level. This encourages us to rethink the design philosophy for Transformers on large graphs, where the global attention is a computation overhead hindering the scalability. We frame the proposed scheme as Simplified Graph Transformers (SGFormer), which is empowered by a simple attention model that can efficiently propagate information among arbitrary nodes in one layer. SGFormer requires none of positional encodings, feature/graph pre-processing or augmented loss. Empirically, SGFormer successfully scales to the web-scale graph ogbn-papers100M and yields up to 141x inference acceleration over SOTA Transformers on medium-sized graphs. Beyond current results, we believe the proposed methodology alone enlightens a new technical path of independent interest for building Transformers on large graphs.

翻译：大规模图上的表示学习因数据点间的相互依赖关系而长期面临挑战。Transformer作为图结构数据的新型基础编码器，凭借其全局注意力机制能够捕获超越邻域节点的全节点对影响，在小规模图上展现出优异性能。然而，现有方法往往沿袭语言和视觉任务中Transformer的设计思想，通过堆叠深层多头注意力机制构建复杂模型。本文关键性论证表明，即使仅使用单层注意力，也能在节点数量从千级到十亿级的节点属性预测基准任务中带来令人惊喜的竞争力。这促使我们重新思考大规模图上Transformer的设计哲学——全局注意力成为阻碍可扩展性的计算瓶颈。我们将所提方案命名为简化图Transformer（SGFormer），其核心在于采用能够高效传播任意节点间信息的单层简易注意力模型。SGFormer无需位置编码、特征/图预处理或增强损失。实验表明，SGFormer成功扩展至web规模图ogbn-papers100M，并在中等规模图上较现有最优Transformer方法实现最高141倍推理加速。除当前成果外，我们相信所提方法论本身为构建大规模图上的Transformer开辟了一条具有独立研究价值的新技术路径。