Dynamic graph embedding has emerged as a very effective technique for addressing diverse temporal graph analytic tasks (i.e., link prediction, node classification, recommender systems, anomaly detection, and graph generation) in various applications. Such temporal graphs exhibit heterogeneous transient dynamics, varying time intervals, and highly evolving node features throughout their evolution. Hence, incorporating long-range dependencies from the historical graph context plays a crucial role in accurately learning their temporal dynamics. In this paper, we develop a graph embedding model with uncertainty quantification, TransformerG2G, by exploiting the advanced transformer encoder to first learn intermediate node representations from its current state ($t$) and previous context (over timestamps [$t-1, t-l$], $l$ is the length of context). Moreover, we employ two projection layers to generate lower-dimensional multivariate Gaussian distributions as each node's latent embedding at timestamp $t$. We consider diverse benchmarks with varying levels of ``novelty" as measured by the TEA plots. Our experiments demonstrate that the proposed TransformerG2G model outperforms conventional multi-step methods and our prior work (DynG2G) in terms of both link prediction accuracy and computational efficiency, especially for high degree of novelty. Furthermore, the learned time-dependent attention weights across multiple graph snapshots reveal the development of an automatic adaptive time stepping enabled by the transformer. Importantly, by examining the attention weights, we can uncover temporal dependencies, identify influential elements, and gain insights into the complex interactions within the graph structure. For example, we identified a strong correlation between attention weights and node degree at the various stages of the graph topology evolution.

翻译：动态图嵌入已成为解决各类应用中多样化时序图分析任务（如链接预测、节点分类、推荐系统、异常检测和图生成）的一种高效技术。这类时序图在整个演化过程中呈现出异质瞬态动力学、不同时间间隔以及高度演化的节点特征。因此，整合历史图上下文的长程依赖性对于准确学习其时序动态性至关重要。本文通过利用先进的Transformer编码器，首先从节点当前状态（$t$）和历史上下文（时间戳区间[$t-1, t-l$]，$l$为上下文长度）中学习中间节点表示，开发了具有不确定性量化能力的图嵌入模型TransformerG2G。此外，我们采用两个投影层生成低维多元高斯分布，作为每个节点在时间戳$t$处的潜在嵌入。我们考虑了TEA图测度下具有不同"新颖度"水平的多样化基准。实验表明，所提出的TransformerG2G模型在链接预测准确率和计算效率方面均优于传统多步方法及我们先前的工作（DynG2G），尤其在高新颖度场景下表现突出。更重要的是，通过多个图快照学习到的时间依赖性注意力权重，揭示了Transformer实现自动自适应时间步长的机制。值得关注的是，通过分析注意力权重，我们能够揭示时序依赖性、识别关键影响因素，并深入理解图结构中的复杂交互模式。例如，我们发现了在图拓扑演化的不同阶段，注意力权重与节点度之间存在强相关性。