In this paper, we present the ``joint pre-training and local re-training'' framework for learning and applying multi-source knowledge graph (KG) embeddings. We are motivated by the fact that different KGs contain complementary information to improve KG embeddings and downstream tasks. We pre-train a large teacher KG embedding model over linked multi-source KGs and distill knowledge to train a student model for a task-specific KG. To enable knowledge transfer across different KGs, we use entity alignment to build a linked subgraph for connecting the pre-trained KGs and the target KG. The linked subgraph is re-trained for three-level knowledge distillation from the teacher to the student, i.e., feature knowledge distillation, network knowledge distillation, and prediction knowledge distillation, to generate more expressive embeddings. The teacher model can be reused for different target KGs and tasks without having to train from scratch. We conduct extensive experiments to demonstrate the effectiveness and efficiency of our framework.

翻译：本文提出了“联合预训练与局部再训练”框架，用于学习及应用多源知识图谱嵌入。我们的动机源于不同知识图谱包含互补信息，可有效提升知识图谱嵌入质量及下游任务性能。我们首先在关联的多源知识图谱上预训练一个大型教师知识图谱嵌入模型，然后通过知识蒸馏训练面向特定任务知识图谱的学生模型。为实现跨知识图谱的知识迁移，我们利用实体对齐构建连接预训练知识图谱与目标知识图谱的关联子图。该关联子图通过三级知识蒸馏（即特征知识蒸馏、网络知识蒸馏和预测知识蒸馏）从教师模型向学生模型进行再训练，以生成更具表达力的嵌入表示。教师模型可重复应用于不同目标知识图谱和任务，无需重新训练。大量实验验证了我们框架的有效性与高效性。