This paper presents a novel transformer architecture for graph representation learning. The core insight of our method is to fully consider the information propagation among nodes and edges in a graph when building the attention module in the transformer blocks. Specifically, we propose a new attention mechanism called Graph Propagation Attention (GPA). It explicitly passes the information among nodes and edges in three ways, i.e. node-to-node, node-to-edge, and edge-to-node, which is essential for learning graph-structured data. On this basis, we design an effective transformer architecture named Graph Propagation Transformer (GPTrans) to further help learn graph data. We verify the performance of GPTrans in a wide range of graph learning experiments on several benchmark datasets. These results show that our method outperforms many state-of-the-art transformer-based graph models with better performance. The code will be released at https://github.com/czczup/GPTrans.

翻译：本文提出了一种新颖的transformer架构，用于图表示学习。该方法的核心思想是在构建transformer模块中的注意力机制时，充分考虑图中节点与边之间的信息传播。具体而言，我们提出了一种名为图传播注意力（Graph Propagation Attention, GPA）的新型注意力机制。该机制显式地通过三种方式在节点与边之间传递信息，即节点-节点、节点-边和边-节点，这对于学习图结构数据至关重要。在此基础上，我们设计了一种高效的transformer架构——图传播变换器（Graph Propagation Transformer, GPTrans），以进一步辅助图数据的学习。我们在多个基准数据集上进行了广泛的图学习实验，验证了GPTrans的性能。结果表明，我们的方法优于许多基于transformer的先进图模型，取得了更优异的表现。代码将在https://github.com/czczup/GPTrans 开源。