Knowledge distillation is the process of transferring knowledge from a more powerful large model (teacher) to a simpler counterpart (student). Numerous current approaches involve the student imitating the knowledge of the teacher directly. However, redundancy still exists in the learned representations through these prevalent methods, which tend to learn each spatial location's features indiscriminately. To derive a more compact representation (concept feature) from the teacher, inspired by human cognition, we suggest an innovative method, termed Generative Denoise Distillation (GDD), where stochastic noises are added to the concept feature of the student to embed them into the generated instance feature from a shallow network. Then, the generated instance feature is aligned with the knowledge of the instance from the teacher. We extensively experiment with object detection, instance segmentation, and semantic segmentation to demonstrate the versatility and effectiveness of our method. Notably, GDD achieves new state-of-the-art performance in the tasks mentioned above. We have achieved substantial improvements in semantic segmentation by enhancing PspNet and DeepLabV3, both of which are based on ResNet-18, resulting in mIoU scores of 74.67 and 77.69, respectively, surpassing their previous scores of 69.85 and 73.20 on the Cityscapes dataset of 20 categories. The source code of GDD is available at https://github.com/ZhgLiu/GDD.

翻译：知识蒸馏是将知识从更强大的大模型（教师）迁移到更简单的对应模型（学生）的过程。当前众多方法要求学生直接模仿教师的知识。然而，这些普遍方法倾向于不加区分地学习每个空间位置的特征，导致所学表示中仍存在冗余。为从教师中提取更紧凑的表示（概念特征），受人类认知启发，我们提出一种创新方法，称为生成式去噪蒸馏（GDD）。该方法将随机噪声添加到学生的概念特征中，使其嵌入由浅层网络生成的实例特征，然后将生成的实例特征与教师的实例知识对齐。我们通过目标检测、实例分割和语义分割的大量实验验证了该方法的多功能性和有效性。值得注意的是，GDD在上述任务中达到了新的最先进性能。通过增强基于ResNet-18的PspNet和DeepLabV3，我们在语义分割中取得了显著提升：在Cityscapes数据集的20个类别上，mIoU得分分别达到74.67和77.69，超越了之前69.85和73.20的成绩。GDD的源代码已公开于https://github.com/ZhgLiu/GDD。