Knowledge distillation has become widely recognized for its ability to transfer knowledge from a large teacher network to a compact and more streamlined student network. Traditional knowledge distillation methods primarily follow a teacher-oriented paradigm that imposes the task of learning the teacher's complex knowledge onto the student network. However, significant disparities in model capacity and architectural design hinder the student's comprehension of the complex knowledge imparted by the teacher, resulting in sub-optimal performance. This paper introduces a novel perspective emphasizing student-oriented and refining the teacher's knowledge to better align with the student's needs, thereby improving knowledge transfer effectiveness. Specifically, we present the Student-Oriented Knowledge Distillation (SoKD), which incorporates a learnable feature augmentation strategy during training to refine the teacher's knowledge of the student dynamically. Furthermore, we deploy the Distinctive Area Detection Module (DAM) to identify areas of mutual interest between the teacher and student, concentrating knowledge transfer within these critical areas to avoid transferring irrelevant information. This customized module ensures a more focused and effective knowledge distillation process. Our approach, functioning as a plug-in, could be integrated with various knowledge distillation methods. Extensive experimental results demonstrate the efficacy and generalizability of our method.

翻译：知识蒸馏因其能够将知识从大型教师网络迁移至紧凑且更精简的学生网络而获得广泛认可。传统的知识蒸馏方法主要遵循教师导向的范式，将学习教师复杂知识的任务强加于学生网络。然而，模型容量和架构设计上的显著差异阻碍了学生对教师所传授复杂知识的理解，导致性能欠佳。本文引入了一种新颖的视角，强调以学生为导向，并精炼教师的知识以更好地适应学生的需求，从而提高知识迁移的有效性。具体而言，我们提出了面向学生的知识蒸馏（SoKD），它在训练过程中引入了一种可学习的特征增强策略，以动态地精炼教师针对学生的知识。此外，我们部署了差异区域检测模块（DAM）来识别教师与学生之间共同关注的区域，将知识迁移集中在这些关键区域内，以避免迁移无关信息。这种定制化的模块确保了知识蒸馏过程更加专注和有效。我们的方法作为一个即插即用模块，可以与多种知识蒸馏方法集成。大量的实验结果证明了我们方法的有效性和泛化能力。