Compared with the feature-based distillation methods, logits distillation can liberalize the requirements of consistent feature dimension between teacher and student networks, while the performance is deemed inferior in face recognition. One major challenge is that the light-weight student network has difficulty fitting the target logits due to its low model capacity, which is attributed to the significant number of identities in face recognition. Therefore, we seek to probe the target logits to extract the primary knowledge related to face identity, and discard the others, to make the distillation more achievable for the student network. Specifically, there is a tail group with near-zero values in the prediction, containing minor knowledge for distillation. To provide a clear perspective of its impact, we first partition the logits into two groups, i.e., Primary Group and Secondary Group, according to the cumulative probability of the softened prediction. Then, we reorganize the Knowledge Distillation (KD) loss of grouped logits into three parts, i.e., Primary-KD, Secondary-KD, and Binary-KD. Primary-KD refers to distilling the primary knowledge from the teacher, Secondary-KD aims to refine minor knowledge but increases the difficulty of distillation, and Binary-KD ensures the consistency of knowledge distribution between teacher and student. We experimentally found that (1) Primary-KD and Binary-KD are indispensable for KD, and (2) Secondary-KD is the culprit restricting KD at the bottleneck. Therefore, we propose a Grouped Knowledge Distillation (GKD) that retains the Primary-KD and Binary-KD but omits Secondary-KD in the ultimate KD loss calculation. Extensive experimental results on popular face recognition benchmarks demonstrate the superiority of proposed GKD over state-of-the-art methods.

翻译：与基于特征的蒸馏方法相比，logits蒸馏可放宽师生网络间特征维度一致性的要求，但该方法在人脸识别中的性能被认为较差。其中一个主要挑战在于，轻量级学生网络因其较低的模型容量难以拟合目标logits，这归因于人脸识别中身份类别的数量庞大。因此，我们试图探测目标logits，提取与人脸身份相关的主要知识并舍弃其他部分，使学生网络的蒸馏更易实现。具体而言，预测结果的尾部存在一组接近零值的logits，其中包含用于蒸馏的次要知识。为清晰阐释其影响，我们首先根据软化预测的累积概率将logits分为两组，即主组与次组。随后，我们将分组后的logits对应的知识蒸馏（KD）损失重组为三个部分：主组-KD、次组-KD和二值-KD。主组-KD旨在从教师网络中蒸馏主要知识，次组-KD旨在细化次要知识但增加了蒸馏难度，二值-KD确保师生网络间知识分布的一致性。实验发现：（1）主组-KD和二值-KD是KD中不可或缺的部分；（2）次组-KD是限制KD瓶颈的关键因素。因此，我们提出分组知识蒸馏（GKD），该方法保留主组-KD和二值-KD，在最终损失计算中舍弃次组-KD。在主流人脸识别基准上的大量实验结果表明，所提GKD方法优于现有最先进方法。