Graph Transformers have emerged as a powerful alternative to Message-Passing Graph Neural Networks (MP-GNNs) to address limitations such as over-squashing of information exchange. However, incorporating graph inductive bias into transformer architectures remains a significant challenge. In this report, we propose the Graph Spectral Token, a novel approach to directly encode graph spectral information, which captures the global structure of the graph, into the transformer architecture. By parameterizing the auxiliary [CLS] token and leaving other tokens representing graph nodes, our method seamlessly integrates spectral information into the learning process. We benchmark the effectiveness of our approach by enhancing two existing graph transformers, GraphTrans and SubFormer. The improved GraphTrans, dubbed GraphTrans-Spec, achieves over 10% improvements on large graph benchmark datasets while maintaining efficiency comparable to MP-GNNs. SubFormer-Spec demonstrates strong performance across various datasets.

翻译：图Transformer已发展成为消息传递图神经网络（MP-GNNs）的有力替代方案，以解决信息交换过度压缩等局限性。然而，将图归纳偏置融入Transformer架构仍是一项重大挑战。本报告提出图频谱令牌，这是一种将捕捉图全局结构的图频谱信息直接编码到Transformer架构中的新颖方法。通过参数化辅助[CLS]令牌并保留其他表示图节点的令牌，我们的方法将频谱信息无缝集成到学习过程中。我们通过增强两种现有图Transformer（GraphTrans和SubFormer）来评估方法的有效性。改进后的GraphTrans（称为GraphTrans-Spec）在大型图基准数据集上实现了超过10%的性能提升，同时保持了与MP-GNNs相当的计算效率。SubFormer-Spec则在多个数据集上展现出强劲性能。