Knowledge distillation (KD) is a simple and successful method to transfer knowledge from a teacher to a student model solely based on functional activity. However, current KD has a few shortcomings: it has recently been shown that this method is unsuitable to transfer simple inductive biases like shift equivariance, struggles to transfer out of domain generalization, and optimization time is magnitudes longer compared to default non-KD model training. To improve these aspects of KD, we propose Hard Augmentations for Robust Distillation (HARD), a generally applicable data augmentation framework, that generates synthetic data points for which the teacher and the student disagree. We show in a simple toy example that our augmentation framework solves the problem of transferring simple equivariances with KD. We then apply our framework in real-world tasks for a variety of augmentation models, ranging from simple spatial transformations to unconstrained image manipulations with a pretrained variational autoencoder. We find that our learned augmentations significantly improve KD performance on in-domain and out-of-domain evaluation. Moreover, our method outperforms even state-of-the-art data augmentations and since the augmented training inputs can be visualized, they offer a qualitative insight into the properties that are transferred from the teacher to the student. Thus HARD represents a generally applicable, dynamically optimized data augmentation technique tailored to improve the generalization and convergence speed of models trained with KD.

翻译：知识蒸馏（KD）是一种简单且成功的方法，它仅基于功能性活动将知识从教师模型迁移至学生模型。然而，当前的KD存在若干缺陷：近期研究表明，该方法不适用于迁移诸如平移等变性等简单归纳偏置，难以实现域外泛化的迁移，且优化时间相较默认的非KD模型训练长数个数量级。为改进KD的这些方面，我们提出“用于鲁棒蒸馏的硬数据增强”（HARD）——一种通用数据增强框架，可生成教师与学生模型产生分歧的合成数据点。我们通过一个简单玩具示例证明，该增强框架解决了利用KD迁移简单等变性的问题。随后，我们将此框架应用于真实世界任务，涵盖从简单空间变换到利用预训练变分自编码器进行无约束图像操作的多种增强模型。实验发现，我们习得的增强方法显著提升了KD在域内和域外评估中的性能。此外，我们的方法甚至优于最先进的数据增强技术，且由于增强后的训练输入可被可视化，它们为教师向学生迁移的属性提供了定性洞察。因此，HARD代表一种通用且动态优化的数据增强技术，专门用于改善经KD训练模型的泛化能力与收敛速度。