Knowledge distillation is a popular technique to transfer knowledge from large teacher models to a small student model. Typically, the student learns to imitate the teacher by minimizing the KL divergence of its output distribution with the teacher's output distribution. In this work, we argue that such a learning objective is sub-optimal because there exists a discrepancy between the teacher's output distribution and the ground truth label distribution. Therefore, forcing the student to blindly imitate the unreliable teacher output distribution leads to inferior performance. To this end, we propose a novel knowledge distillation objective PTLoss by first representing the vanilla KL-based distillation loss function via a Maclaurin series and then perturbing the leading-order terms in this series. This perturbed loss implicitly transforms the original teacher into a proxy teacher with a distribution closer to the ground truth distribution. We establish the theoretical connection between this "distribution closeness" and the student model generalizability, which enables us to select the PTLoss's perturbation coefficients in a principled way. Extensive experiments on five datasets demonstrate PTLoss can significantly improve the distillation effectiveness for teachers of various scales.

翻译：知识蒸馏是一种将知识从大型教师模型迁移到小型学生模型的流行技术。通常，学生通过最小化其输出分布与教师输出分布之间的KL散度来学习模仿教师。然而，本文认为这种学习目标并非最优，因为教师的输出分布与真实标签分布之间存在差异。因此，强迫学生盲目模仿不可靠的教师输出分布会导致性能不佳。为此，我们提出了一种新颖的知识蒸馏目标PTLoss，首先将基于标准KL的蒸馏损失函数通过麦克劳林级数展开，然后对该级数中的主导项进行扰动。这种扰动损失隐式地将原始教师转换为一个分布更接近真实标签分布的代理教师。我们建立了这种“分布接近性”与学生模型泛化能力之间的理论联系，从而能够以规范的方式选择PTLoss的扰动系数。在五个数据集上的大量实验表明，PTLoss能够显著提升不同规模教师模型的蒸馏效果。