In Knowledge Distillation, the teacher is generally much larger than the student, making the solution of the teacher likely to be difficult for the student to learn. To ease the mimicking difficulty, we introduce a triplet knowledge distillation mechanism named TriKD. Besides teacher and student, TriKD employs a third role called anchor model. Before distillation begins, the pre-trained anchor model delimits a subspace within the full solution space of the target problem. Solutions within the subspace are expected to be easy targets that the student could mimic well. Distillation then begins in an online manner, and the teacher is only allowed to express solutions within the aforementioned subspace. Surprisingly, benefiting from accurate but easy-to-mimic hints, the student can finally perform well. After the student is well trained, it can be used as the new anchor for new students, forming a curriculum learning strategy. Our experiments on image classification and face recognition with various models clearly demonstrate the effectiveness of our method. Furthermore, the proposed TriKD is also effective in dealing with the overfitting issue. Moreover, our theoretical analysis supports the rationality of our triplet distillation.

翻译：在知识蒸馏中，教师模型通常远大于学生模型，导致教师模型的解对学生而言难以学习。为缓解模仿难度，我们提出了一种名为TriKD的三元知识蒸馏机制。除教师和学生外，TriKD引入了第三个角色——锚定模型。蒸馏开始前，预训练的锚定模型在目标问题的完整解空间中界定出一个子空间，该子空间内的解被视为学生能够有效模仿的简易目标。随后，蒸馏以在线方式启动，教师模型仅允许在前述子空间内表达解。令人惊讶的是，凭借精准且易于模仿的提示，学生模型最终能取得良好表现。在学生模型充分训练后，它可作为新学生的锚定模型，形成课程学习策略。我们在图像分类和人脸识别任务上使用多种模型进行的实验，清晰证明了本方法的有效性。此外，所提出的TriKD在处理过拟合问题上同样有效。最后，我们的理论分析支持了三元蒸馏的合理性。