Mixed Sample Regularization (MSR), such as MixUp or CutMix, is a powerful data augmentation strategy to generalize convolutional neural networks. Previous empirical analysis has illustrated an orthogonal performance gain between MSR and conventional offline Knowledge Distillation (KD). To be more specific, student networks can be enhanced with the involvement of MSR in the training stage of sequential distillation. Yet, the interplay between MSR and online knowledge distillation, where an ensemble of peer students learn mutually from each other, remains unexplored. To bridge the gap, we make the first attempt at incorporating CutMix into online distillation, where we empirically observe a significant improvement. Encouraged by this fact, we propose an even stronger MSR specifically for online distillation, named as Cut\textsuperscript{n}Mix. Furthermore, a novel online distillation framework is designed upon Cut\textsuperscript{n}Mix, to enhance the distillation with feature level mutual learning and a self-ensemble teacher. Comprehensive evaluations on CIFAR10 and CIFAR100 with six network architectures show that our approach can consistently outperform state-of-the-art distillation methods.

翻译：混合样本正则化（MSR），如MixUp或CutMix，是一种强大的数据增强策略，用于泛化卷积神经网络。先前的实证分析表明，MSR与传统的离线知识蒸馏（KD）之间存在正交的性能提升。具体来说，在顺序蒸馏的训练阶段引入MSR可以增强学生网络。然而，MSR与在线知识蒸馏（即一组同级学生网络相互学习）之间的相互作用尚未被探索。为填补这一空白，我们首次尝试将CutMix引入在线蒸馏，并在实证中观察到显著的改进。受此启发，我们提出了一种专门针对在线蒸馏的更强大的MSR方法，命名为CutⁿMix。此外，基于CutⁿMix设计了一种新颖的在线蒸馏框架，通过特征级互相学习和自集成教师来增强蒸馏效果。在CIFAR10和CIFAR100数据集上使用六种网络架构进行的全面评估表明，我们的方法能够持续优于最先进的蒸馏方法。