In this paper, we propose a feature affinity (FA) assisted knowledge distillation (KD) method to improve quantization-aware training of deep neural networks (DNN). The FA loss on intermediate feature maps of DNNs plays the role of teaching middle steps of a solution to a student instead of only giving final answers in the conventional KD where the loss acts on the network logits at the output level. Combining logit loss and FA loss, we found that the quantized student network receives stronger supervision than from the labeled ground-truth data. The resulting FAQD is capable of compressing model on label-free data, which brings immediate practical benefits as pre-trained teacher models are readily available and unlabeled data are abundant. In contrast, data labeling is often laborious and expensive. Finally, we propose a fast feature affinity (FFA) loss that accurately approximates FA loss with a lower order of computational complexity, which helps speed up training for high resolution image input.

翻译：本文提出一种特征亲和力（FA）辅助的知识蒸馏（KD）方法，以改进深度神经网络（DNN）的量化感知训练。与仅在输出层使用网络logits进行损失计算的常规KD不同，FA损失作用于DNN中间特征图，其角色相当于向学生模型传授解题步骤而非仅提供最终答案。通过结合logit损失与FA损失，我们发现量化后的学生网络能获得比标注真实数据更强的监督信号。由此产生的FAQD方法可在无标签数据上实现模型压缩——由于预训练教师模型易于获取且无标签数据极其丰富，该方法具备直接的实际效益。相比之下，数据标注往往需要耗费大量人力与成本。最后，我们提出快速特征亲和力（FFA）损失，其以更低计算复杂度精确近似FA损失，有助于加速高分辨率图像输入的训练过程。