We propose Cross-Attention-based Non-local Knowledge Distillation (CanKD), a novel feature-based knowledge distillation framework that leverages cross-attention mechanisms to enhance the knowledge transfer process. Unlike traditional self-attention-based distillation methods that align teacher and student feature maps independently, CanKD enables each pixel in the student feature map to dynamically consider all pixels in the teacher feature map. This non-local knowledge transfer more thoroughly captures pixel-wise relationships, improving feature representation learning. Our method introduces only an additional loss function to achieve superior performance compared with existing attention-guided distillation methods. Extensive experiments on object detection and image segmentation tasks demonstrate that CanKD outperforms state-of-the-art feature and hybrid distillation methods. These experimental results highlight CanKD's potential as a new paradigm for attention-guided distillation in computer vision tasks. Code is available at https://github.com/tori-hotaru/CanKD

翻译：我们提出了基于交叉注意力的非局部知识蒸馏（CanKD），这是一种新颖的基于特征的知识蒸馏框架，利用交叉注意力机制来增强知识传递过程。与传统的基于自注意力的蒸馏方法（独立对齐教师和学生特征图）不同，CanKD使学生特征图中的每个像素能够动态地考虑教师特征图中的所有像素。这种非局部知识传递更全面地捕捉像素级关系，从而改善特征表示学习。我们的方法仅引入一个额外的损失函数，即可在性能上优于现有的注意力引导蒸馏方法。在目标检测和图像分割任务上的大量实验表明，CanKD超越了当前最先进的基于特征和混合蒸馏方法。这些实验结果突显了CanKD作为计算机视觉任务中注意力引导蒸馏新范式的潜力。代码可在 https://github.com/tori-hotaru/CanKD 获取。