Graph transformers have emerged as a promising architecture for a variety of graph learning and representation tasks. Despite their successes, though, it remains challenging to scale graph transformers to large graphs while maintaining accuracy competitive with message-passing networks. In this paper, we introduce Exphormer, a framework for building powerful and scalable graph transformers. Exphormer consists of a sparse attention mechanism based on two mechanisms: virtual global nodes and expander graphs, whose mathematical characteristics, such as spectral expansion, pseduorandomness, and sparsity, yield graph transformers with complexity only linear in the size of the graph, while allowing us to prove desirable theoretical properties of the resulting transformer models. We show that incorporating \textsc{Exphormer} into the recently-proposed GraphGPS framework produces models with competitive empirical results on a wide variety of graph datasets, including state-of-the-art results on three datasets. We also show that \textsc{Exphormer} can scale to datasets on larger graphs than shown in previous graph transformer architectures. Code can be found at https://github.com/hamed1375/Exphormer.

翻译：图Transformer已成为多种图学习与表征任务中的一类有前景的架构。然而，尽管取得了成功，如何将图Transformer扩展到大规模图，同时保持与消息传递网络相竞争的性能，仍是一个挑战。本文提出了Exphormer，一个用于构建强大且可扩展的图Transformer的框架。Exphormer的稀疏注意力机制基于两类组件：虚拟全局节点和展开图（expander graphs）。其数学特性（如谱展开、伪随机性和稀疏性）使得图Transformer的复杂度仅与图的规模呈线性关系，同时我们能够证明所得Transformer模型具有理想的理论性质。实验表明，将Exphormer集成到近期提出的GraphGPS框架中，能在多种图数据集上取得具有竞争力的实证结果，包括在三个数据集上达到当前最优性能。我们还证明，Exphormer能够处理比以往图Transformer架构所展示的更大规模的图数据集。代码见https://github.com/hamed1375/Exphormer。