Transformer models have recently gained popularity in graph representation learning as they have the potential to learn complex relationships beyond the ones captured by regular graph neural networks. The main research question is how to inject the structural bias of graphs into the transformer architecture, and several proposals have been made for undirected molecular graphs and, recently, also for larger network graphs. In this paper, we study transformers over directed acyclic graphs (DAGs) and propose architecture adaptations tailored to DAGs: (1) An attention mechanism that is more efficient than the regular quadratic complexity of transformers and at the same time faithfully captures the DAG structure, and (2) a positional encoding of the DAG's partial order, complementing the former. We rigorously evaluate our framework in ablation studies and show that it is effective in improving different kinds of baseline transformers over various types of data, in experiments ranging from classifying source code graphs to nodes in self-citation networks. In particular, our proposal makes (graph) transformers competitive to or outperform graph neural networks tailored to DAGs.

翻译：Transformer模型近年来在图表示学习中受到广泛关注，因其具有学习超越常规图神经网络所捕获复杂关系的潜力。核心研究问题在于如何将图的结构偏置注入Transformer架构，目前已有多个针对无向分子图的方案提出，最近也出现了面向大规模网络图的方案。本文研究基于有向无环图（DAG）的Transformer模型，并提出针对DAG的自适应架构改进：（1）一种比常规Transformer二次复杂度更高效、同时能精确捕获DAG结构的注意力机制；（2）一种对DAG偏序关系的位置编码方法，作为前者的补充。我们通过消融实验严格评估了所提框架，并在从源代码图分类到自引网络节点分类等多样化数据实验中证明，该方法能有效改进各类基线Transformer模型。特别地，我们的方案使得（图）Transformer在与专为DAG设计的图神经网络竞争中具有优势，甚至能超越后者。