Many existing studies on knowledge distillation have focused on methods in which a student model mimics a teacher model well. Simply imitating the teacher's knowledge, however, is not sufficient for the student to surpass that of the teacher. We explore a method to harness the knowledge of other students to complement the knowledge of the teacher. We propose deep collective knowledge distillation for model compression, called DCKD, which is a method for training student models with rich information to acquire knowledge from not only their teacher model but also other student models. The knowledge collected from several student models consists of a wealth of information about the correlation between classes. Our DCKD considers how to increase the correlation knowledge of classes during training. Our novel method enables us to create better performing student models for collecting knowledge. This simple yet powerful method achieves state-of-the-art performances in many experiments. For example, for ImageNet, ResNet18 trained with DCKD achieves 72.27\%, which outperforms the pretrained ResNet18 by 2.52\%. For CIFAR-100, the student model of ShuffleNetV1 with DCKD achieves 6.55\% higher top-1 accuracy than the pretrained ShuffleNetV1.

翻译：现有关于知识蒸馏的研究多集中在学生模型如何良好地模仿教师模型的方法上。然而，仅通过模仿教师的知识并不足以使学生模型超越教师模型。我们探索了一种利用其他学生模型知识来补充教师模型知识的方法。我们提出了一种面向模型压缩的深度集体知识蒸馏方法，称为DCKD，该方法用于训练具备丰富信息的学生模型，使其不仅从教师模型获取知识，还能从其他学生模型获取知识。从多个学生模型中收集的知识包含了关于类别间相关性的丰富信息。我们的DCKD方法考虑了如何在训练过程中增强类别间的相关性知识。这一新颖方法使我们能够创建性能更优的学生模型来收集知识。这种简单而强大的方法在多项实验中达到了最先进的性能。例如，在ImageNet数据集上，采用DCKD训练的ResNet18达到了72.27%的准确率，比预训练的ResNet18高出2.52%。在CIFAR-100数据集上，采用DCKD的ShuffleNetV1学生模型的top-1准确率比预训练的ShuffleNetV1高出6.55%。