Representation learning on networks aims to derive a meaningful vector representation for each node, thereby facilitating downstream tasks such as link prediction, node classification, and node clustering. In heterogeneous text-rich networks, this task is more challenging due to (1) presence or absence of text: Some nodes are associated with rich textual information, while others are not; (2) diversity of types: Nodes and edges of multiple types form a heterogeneous network structure. As pretrained language models (PLMs) have demonstrated their effectiveness in obtaining widely generalizable text representations, a substantial amount of effort has been made to incorporate PLMs into representation learning on text-rich networks. However, few of them can jointly consider heterogeneous structure (network) information as well as rich textual semantic information of each node effectively. In this paper, we propose Heterformer, a Heterogeneous Network-Empowered Transformer that performs contextualized text encoding and heterogeneous structure encoding in a unified model. Specifically, we inject heterogeneous structure information into each Transformer layer when encoding node texts. Meanwhile, Heterformer is capable of characterizing node/edge type heterogeneity and encoding nodes with or without texts. We conduct comprehensive experiments on three tasks (i.e., link prediction, node classification, and node clustering) on three large-scale datasets from different domains, where Heterformer outperforms competitive baselines significantly and consistently.

翻译：网络表示学习旨在为每个节点生成有意义的向量表示，从而促进链路预测、节点分类和节点聚类等下游任务。在异构文本丰富网络中，该任务更具挑战性，原因在于：（1）文本存在与否：部分节点关联丰富的文本信息，而其他节点则没有；（2）类型多样性：多类型节点和边构成异构网络结构。由于预训练语言模型（PLMs）在获取广泛可泛化的文本表示方面已展现出有效性，大量研究致力于将PLMs融入文本丰富网络的表示学习。然而，现有方法鲜有能有效联合考虑异构结构（网络）信息与各节点丰富的文本语义信息。本文提出Heterformer，一种异构网络增强型Transformer，可在统一模型中执行上下文文本编码和异构结构编码。具体而言，我们在编码节点文本时将异构结构信息注入每个Transformer层。同时，Heterformer能够刻画节点/边类型的异构性，并对有文本或无文本的节点进行编码。我们在三个不同领域的大规模数据集上，针对三个任务（即链路预测、节点分类和节点聚类）进行了全面实验，结果表明Heterformer显著且一致地优于竞争基线方法。