Knowledge distillation (KD) remains challenging due to the opaque nature of the knowledge transfer process from a Teacher to a Student, making it difficult to address certain issues related to KD. To address this, we proposed UniCAM, a novel gradient-based visual explanation method, which effectively interprets the knowledge learned during KD. Our experimental results demonstrate that with the guidance of the Teacher's knowledge, the Student model becomes more efficient, learning more relevant features while discarding those that are not relevant. We refer to the features learned with the Teacher's guidance as distilled features and the features irrelevant to the task and ignored by the Student as residual features. Distilled features focus on key aspects of the input, such as textures and parts of objects. In contrast, residual features demonstrate more diffused attention, often targeting irrelevant areas, including the backgrounds of the target objects. In addition, we proposed two novel metrics: the feature similarity score (FSS) and the relevance score (RS), which quantify the relevance of the distilled knowledge. Experiments on the CIFAR10, ASIRRA, and Plant Disease datasets demonstrate that UniCAM and the two metrics offer valuable insights to explain the KD process.

翻译：知识蒸馏（KD）因其从教师模型到学生模型的知识传递过程不透明而仍具挑战性，这使得解决与KD相关的某些问题变得困难。为此，我们提出了UniCAM——一种新颖的基于梯度的可视化解释方法，它能有效阐释KD过程中习得的知识。实验结果表明，在教师模型知识的引导下，学生模型变得更高效，能够学习更多相关特征，同时舍弃不相关的特征。我们将受教师引导习得的特征称为蒸馏特征，将与任务无关且被学生模型忽略的特征称为残差特征。蒸馏特征集中于输入的关键方面，例如纹理和物体局部。相比之下，残差特征表现出更分散的注意力，常聚焦于不相关区域，包括目标物体的背景。此外，我们提出了两个新颖的度量指标：特征相似度分数（FSS）和相关性分数（RS），用以量化蒸馏知识的相关性。在CIFAR10、ASIRRA和植物病害数据集上的实验表明，UniCAM及这两个度量指标为解释KD过程提供了有价值的见解。