The representation gap between teacher and student is an emerging topic in knowledge distillation (KD). To reduce the gap and improve the performance, current methods often resort to complicated training schemes, loss functions, and feature alignments, which are task-specific and feature-specific. In this paper, we state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature, and propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models. Our approach is based on the observation that student features typically contain more noises than teacher features due to the smaller capacity of student model. To address this, we propose to denoise student features using a diffusion model trained by teacher features. This allows us to perform better distillation between the refined clean feature and teacher feature. Additionally, we introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost and an adaptive noise matching module to improve the denoising performance. Extensive experiments demonstrate that DiffKD is effective across various types of features and achieves state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at https://github.com/hunto/DiffKD.

翻译：师生表示差异是知识蒸馏（KD）中一个新兴的研究方向。为缩小这一差异并提升性能，现有方法通常采用复杂的训练策略、损失函数及特征对齐操作，这些方法往往具有任务特异性和特征特异性。本文指出，这些方法的本质是去除特征中的噪声信息并提取有价值的信息进行蒸馏，为此我们提出一种名为DiffKD的新型知识蒸馏方法，旨在利用扩散模型显式地实现特征降噪与匹配。我们的方法基于以下发现：由于学生模型容量较小，其学生特征通常包含比教师特征更多的噪声。针对这一问题，我们提出利用教师特征训练的扩散模型对学生特征进行降噪处理，从而在精炼后的干净特征与教师特征之间实现更有效的蒸馏。此外，我们引入了一种轻量级扩散模型（含线性自编码器）以降低计算成本，并设计自适应噪声匹配模块提升降噪性能。大量实验表明，DiffKD在多种特征类型上均表现有效，在图像分类、目标检测和语义分割任务中持续达到最先进性能。代码已开源至https://github.com/hunto/DiffKD。