Traditional knowledge graph embedding (KGE) methods typically require preserving the entire knowledge graph (KG) with significant training costs when new knowledge emerges. To address this issue, the continual knowledge graph embedding (CKGE) task has been proposed to train the KGE model by learning emerging knowledge efficiently while simultaneously preserving decent old knowledge. However, the explicit graph structure in KGs, which is critical for the above goal, has been heavily ignored by existing CKGE methods. On the one hand, existing methods usually learn new triples in a random order, destroying the inner structure of new KGs. On the other hand, old triples are preserved with equal priority, failing to alleviate catastrophic forgetting effectively. In this paper, we propose a competitive method for CKGE based on incremental distillation (IncDE), which considers the full use of the explicit graph structure in KGs. First, to optimize the learning order, we introduce a hierarchical strategy, ranking new triples for layer-by-layer learning. By employing the inter- and intra-hierarchical orders together, new triples are grouped into layers based on the graph structure features. Secondly, to preserve the old knowledge effectively, we devise a novel incremental distillation mechanism, which facilitates the seamless transfer of entity representations from the previous layer to the next one, promoting old knowledge preservation. Finally, we adopt a two-stage training paradigm to avoid the over-corruption of old knowledge influenced by under-trained new knowledge. Experimental results demonstrate the superiority of IncDE over state-of-the-art baselines. Notably, the incremental distillation mechanism contributes to improvements of 0.2%-6.5% in the mean reciprocal rank (MRR) score.

翻译：传统知识图谱嵌入方法通常需要保留完整知识图谱，在新知识出现时需耗费大量训练成本。为此，持续知识图谱嵌入任务被提出，旨在通过高效学习新知识的同时妥善保留旧知识，对知识图谱嵌入模型进行训练。然而，现有持续知识图谱嵌入方法严重忽视了知识图谱中显式图结构对上述目标的关键作用。一方面，现有方法通常以随机顺序学习新三元组，破坏了新知识图谱的内在结构；另一方面，旧三元组被等优先级保留，未能有效缓解灾难性遗忘。本文提出基于增量蒸馏的竞争性持续知识图谱嵌入方法IncDE，该方法充分考虑了知识图谱中显式图结构的利用。首先，为优化学习顺序，我们引入层次化策略，将新三元组排序后进行逐层学习。通过联合运用层间与层内排序，新三元组基于图结构特征被划分为不同层级。其次，为有效保留旧知识，我们设计新型增量蒸馏机制，促进实体表示从前一层无缝迁移至下一层，从而强化旧知识保持。最后，采用两阶段训练范式，避免未充分训练的新知识过度破坏旧知识。实验结果表明，IncDE方法在性能上优于现有最优基线。值得注意的是，增量蒸馏机制使平均倒数秩评分提升0.2%-6.5%。