In knowledge distillation, the knowledge from the teacher model is often too complex for the student model to thoroughly process. However, good teachers in real life always simplify complex material before teaching it to students. Inspired by this fact, we propose student-friendly knowledge distillation (SKD) to simplify teacher output into new knowledge representations, which makes the learning of the student model easier and more effective. SKD contains a softening processing and a learning simplifier. First, the softening processing uses the temperature hyperparameter to soften the output logits of the teacher model, which simplifies the output to some extent and makes it easier for the learning simplifier to process. The learning simplifier utilizes the attention mechanism to further simplify the knowledge of the teacher model and is jointly trained with the student model using the distillation loss, which means that the process of simplification is correlated with the training objective of the student model and ensures that the simplified new teacher knowledge representation is more suitable for the specific student model. Furthermore, since SKD does not change the form of the distillation loss, it can be easily combined with other distillation methods that are based on the logits or features of intermediate layers to enhance its effectiveness. Therefore, SKD has wide applicability. The experimental results on the CIFAR-100 and ImageNet datasets show that our method achieves state-of-the-art performance while maintaining high training efficiency.

翻译：在知识蒸馏中，教师模型的知识往往过于复杂，学生模型难以全面处理。然而，现实生活中的优秀教师总是在授课前将复杂材料简化。受此启发，我们提出面向学生友好的知识蒸馏（SKD），通过将教师输出简化为新的知识表示，使学生模型的学习更轻松且更高效。SKD包含软化处理和学习简化器两个部分。首先，软化处理利用温度超参数对教师模型的输出logits进行软化，在一定程度上简化输出，便于学习简化器处理。学习简化器通过注意力机制进一步简化教师模型的知识，并与学生模型联合训练，利用蒸馏损失进行优化。这意味着简化过程与学生模型的训练目标相关联，从而确保简化后的教师知识表示更适应特定学生模型。此外，由于SKD未改变蒸馏损失的形式，它可以轻松与其他基于logits或中间层特征的蒸馏方法结合以增强效果，因此具有广泛的适用性。在CIFAR-100和ImageNet数据集上的实验结果表明，我们的方法在保持高训练效率的同时达到了最先进的性能。