Recently, Transformers for graph representation learning have become increasingly popular, achieving state-of-the-art performance on a wide-variety of datasets, either alone or in combination with message-passing graph neural networks (MP-GNNs). Infusing graph inductive-biases in the innately structure-agnostic transformer architecture in the form of structural or positional encodings (PEs) is key to achieving these impressive results. However, designing such encodings is tricky and disparate attempts have been made to engineer such encodings including Laplacian eigenvectors, relative random-walk probabilities (RRWP), spatial encodings, centrality encodings, edge encodings etc. In this work, we argue that such encodings may not be required at all, provided the attention mechanism itself incorporates information about the graph structure. We introduce Eigenformer, a Graph Transformer employing a novel spectrum-aware attention mechanism cognizant of the Laplacian spectrum of the graph, and empirically show that it achieves performance comparable to SOTA Graph Transformers on a number of standard GNN benchmark datasets, even surpassing the SOTA on some datasets. The simpler attention mechanism also allows us to train wider and deeper models for a given parameter budget.

翻译：近期，用于图表示学习的变换器（Transformers）在众多数据集上单独或与消息传递图神经网络（MP-GNNs）结合取得了最先进的性能。作为本质结构无关的变换器架构，通过结构或位置编码（PEs）注入图归纳偏置是实现这些卓越性能的关键。然而，设计此类编码颇具挑战性，研究者们已尝试多种方案，包括拉普拉斯特征向量、相对随机游走概率（RRWP）、空间编码、中心性编码、边编码等。本文提出，只要注意力机制本身能整合图结构信息，这类编码可能完全不需要。我们引入Eigenformer——一种采用新型频谱感知注意力机制的图变换器，该机制能识别图的拉普拉斯频谱，实验表明其在多个标准GNN基准数据集上性能可媲美最先进的图变换器，甚至在某些数据集上超越现有最优水平。更简洁的注意力机制还允许我们在给定参数预算下训练更宽更深的模型。