Previous knowledge distillation methods have shown their impressive performance on model compression tasks, however, it is hard to explain how the knowledge they transferred helps to improve the performance of the student network. In this work, we focus on proposing a knowledge distillation method that has both high interpretability and competitive performance. We first revisit the structure of mainstream CNN models and reveal that possessing the capacity of identifying class discriminative regions of input is critical for CNN to perform classification. Furthermore, we demonstrate that this capacity can be obtained and enhanced by transferring class activation maps. Based on our findings, we propose class attention transfer based knowledge distillation (CAT-KD). Different from previous KD methods, we explore and present several properties of the knowledge transferred by our method, which not only improve the interpretability of CAT-KD but also contribute to a better understanding of CNN. While having high interpretability, CAT-KD achieves state-of-the-art performance on multiple benchmarks. Code is available at: https://github.com/GzyAftermath/CAT-KD.

翻译：以往的知识蒸馏方法在模型压缩任务上表现出令人瞩目的性能，然而，所迁移的知识如何帮助提升学生网络性能这一机制难以解释。本研究旨在提出一种兼具高可解释性与竞争性能的知识蒸馏方法。我们首先重新审视主流CNN模型的结构，揭示出具备识别输入中类别判别性区域的能力对CNN执行分类任务至关重要。进一步地，我们证明这种能力可通过迁移类激活图来获取并增强。基于上述发现，我们提出基于类别注意力迁移的知识蒸馏（CAT-KD）。与以往知识蒸馏方法不同，我们探索并展示了本方法所迁移知识的若干特性，这不仅提升了CAT-KD的可解释性，也有助于更深入地理解CNN。在保持高可解释性的同时，CAT-KD在多个基准测试中达到了最先进的性能。代码开源地址：https://github.com/GzyAftermath/CAT-KD。