Recently, Graph Transformers have emerged as a promising solution to alleviate the inherent limitations of Graph Neural Networks (GNNs) and enhance graph representation performance. Unfortunately, Graph Transformers are computationally expensive due to the quadratic complexity inherent in self-attention when applied over large-scale graphs, especially for node tasks. In contrast, spiking neural networks (SNNs), with event-driven and binary spikes properties, can perform energy-efficient computation. In this work, we propose a novel insight into integrating SNNs with Graph Transformers and design a Spiking Graph Attention (SGA) module. The matrix multiplication is replaced by sparse addition and mask operations. The linear complexity enables all-pair node interactions on large-scale graphs with limited GPU memory. To our knowledge, our work is the first attempt to introduce SNNs into Graph Transformers. Furthermore, we design SpikeGraphormer, a Dual-branch architecture, combining a sparse GNN branch with our SGA-driven Graph Transformer branch, which can simultaneously perform all-pair node interactions and capture local neighborhoods. SpikeGraphormer consistently outperforms existing state-of-the-art approaches across various datasets and makes substantial improvements in training time, inference time, and GPU memory cost (10 ~ 20x lower than vanilla self-attention). It also performs well in cross-domain applications (image and text classification). We release our code at https://github.com/PHD-lanyu/SpikeGraphormer.

翻译：近年来，图Transformer作为缓解图神经网络（GNN）固有局限性并提升图表示性能的有前景方案而兴起。然而，图Transformer在处理大规模图时（尤其是节点任务），由于自注意力机制固有的二次复杂度，计算成本极高。相比之下，脉冲神经网络（SNN）凭借事件驱动和二进制脉冲特性，可实现能效计算。本文提出将SNN与图Transformer融合的新颖思路，并设计了脉冲图注意力（SGA）模块。我们将矩阵乘法替换为稀疏加法与掩码操作，其线性复杂度使得在有限GPU内存下仍能对大规模图执行全节点对交互。据我们所知，这是首次将SNN引入图Transformer的尝试。此外，我们设计了双分支架构SpikeGraphormer，融合稀疏GNN分支与SGA驱动的图Transformer分支，可同时实现全节点对交互与局部邻域捕获。在多个数据集上，SpikeGraphormer持续超越现有最优方法，并在训练时间、推理时间和GPU内存开销方面实现显著提升（比原始自注意力机制降低10~20倍）。该方法在跨领域应用（图像与文本分类）中同样表现优异。代码已开源至https://github.com/PHD-lanyu/SpikeGraphormer。