Transformers have achieved remarkable performance in widespread fields, including natural language processing, computer vision and graph mining. However, vanilla Transformer architectures have not yielded promising improvements in the Knowledge Graph (KG) representations, where the translational distance paradigm dominates this area. Note that vanilla Transformer architectures struggle to capture the intrinsically heterogeneous structural and semantic information of knowledge graphs. To this end, we propose a new variant of Transformer for knowledge graph representations dubbed Relphormer. Specifically, we introduce Triple2Seq which can dynamically sample contextualized sub-graph sequences as the input to alleviate the heterogeneity issue. We propose a novel structure-enhanced self-attention mechanism to encode the relational information and keep the semantic information within entities and relations. Moreover, we utilize masked knowledge modeling for general knowledge graph representation learning, which can be applied to various KG-based tasks including knowledge graph completion, question answering, and recommendation. Experimental results on six datasets show that Relphormer can obtain better performance compared with baselines. Code is available in https://github.com/zjunlp/Relphormer.

翻译：Transformer在自然语言处理、计算机视觉和图挖掘等广泛领域取得了显著性能。然而，标准Transformer架构在知识图谱表示领域尚未带来显著改进，该领域仍以平移距离范式为主导。值得注意的是，标准Transformer架构难以捕捉知识图谱中本质上异质性的结构和语义信息。为此，我们提出了一种用于知识图谱表示的新型Transformer变体，名为Relphormer。具体而言，我们引入了Triple2Seq，该方法可以动态采样上下文相关的子图序列作为输入，以缓解异质性问题。我们提出了一种新颖的结构增强自注意力机制，用于编码关系信息并保持实体和关系内的语义信息。此外，我们利用掩码知识建模进行通用知识图谱表示学习，该方法可应用于多种基于知识图谱的任务，包括知识图谱补全、问答和推荐。在六个数据集上的实验结果表明，与基线方法相比，Relphormer能够获得更优性能。代码可在https://github.com/zjunlp/Relphormer获取。