Knowledge distillation (KD) is an effective training strategy to improve the lightweight student models under the guidance of cumbersome teachers. However, the large architecture difference across the teacher-student pairs limits the distillation gains. In contrast to previous adaptive distillation methods to reduce the teacher-student gap, we explore a novel training-free framework to search for the best student architectures for a given teacher. Our work first empirically show that the optimal model under vanilla training cannot be the winner in distillation. Secondly, we find that the similarity of feature semantics and sample relations between random-initialized teacher-student networks have good correlations with final distillation performances. Thus, we efficiently measure similarity matrixs conditioned on the semantic activation maps to select the optimal student via an evolutionary algorithm without any training. In this way, our student architecture search for Distillation WithOut Training (DisWOT) significantly improves the performance of the model in the distillation stage with at least 180$\times$ training acceleration. Additionally, we extend similarity metrics in DisWOT as new distillers and KD-based zero-proxies. Our experiments on CIFAR, ImageNet and NAS-Bench-201 demonstrate that our technique achieves state-of-the-art results on different search spaces. Our project and code are available at https://lilujunai.github.io/DisWOT-CVPR2023/.

翻译：知识蒸馏（KD）是一种有效的训练策略，能在笨重的教师模型指导下提升轻量级学生模型的性能。然而，师生模型间的巨大架构差异限制了蒸馏效果的提升。与以往旨在缩小师生差距的自适应蒸馏方法不同，我们探索了一种新颖的无训练框架，为给定教师模型搜索最佳学生架构。我们的工作首先通过实验表明，原始训练下的最优模型在蒸馏中并非胜者。其次，我们发现随机初始化的师生网络间的特征语义相似度和样本关系相似度与最终蒸馏性能具有良好的相关性。因此，我们基于语义激活图高效度量相似矩阵，通过进化算法无需任何训练即可选择最优学生模型。通过这种方式，我们的"无训练蒸馏学生架构搜索"（DisWOT）方法至少实现了180倍的训练加速，同时显著提升了模型在蒸馏阶段的性能。此外，我们将DisWOT中的相似度度量扩展为新的蒸馏器及基于KD的零代理指标。在CIFAR、ImageNet和NAS-Bench-201上的实验表明，我们的技术在不同搜索空间上均取得了最先进的结果。相关项目与代码已开源：https://lilujunai.github.io/DisWOT-CVPR2023/。