In knowledge distillation, since a single, omnipotent teacher network cannot solve all problems, multiple teacher-based knowledge distillations have been studied recently. However, sometimes their improvements are not as good as expected because some immature teachers may transfer the false knowledge to the student. In this paper, to overcome this limitation and take the efficacy of the multiple networks, we divide the multiple networks into teacher and student groups, respectively. That is, the student group is a set of immature networks that require learning the teacher's knowledge, while the teacher group consists of the selected networks that are capable of teaching successfully. We propose our online role change strategy where the top-ranked networks in the student group are able to promote to the teacher group at every iteration. After training the teacher group using the error samples of the student group to refine the teacher group's knowledge, we transfer the collaborative knowledge from the teacher group to the student group successfully. We verify the superiority of the proposed method on CIFAR-10, CIFAR-100, and ImageNet which achieves high performance. We further show the generality of our method with various backbone architectures such as ResNet, WRN, VGG, Mobilenet, and Shufflenet.

翻译：在知识蒸馏中，由于单一的万能教师网络无法解决所有问题，近年来已有多教师知识蒸馏的研究。然而，其改进效果有时不如预期，因为部分不成熟的教师可能向学生传递错误知识。为克服这一局限并充分发挥多网络效能，本文提出将多个网络分别划分为教师组与学生组。即学生组由需要学习教师知识的一组不成熟网络组成，而教师组则由具备成功教学能力的选定网络构成。我们提出在线角色交换策略，使学生组中排名靠前的网络能够在每次迭代中晋升至教师组。通过利用学生组的错误样本训练教师组以完善其知识后，我们将教师组的协作知识成功迁移至学生组。我们在CIFAR-10、CIFAR-100和ImageNet上验证了所提方法的优越性，其取得了高性能表现。此外，我们通过ResNet、WRN、VGG、MobileNet和ShuffleNet等多种骨干架构展示了方法的通用性。