Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of deep models trained with SGD. In this work, we report a previously unobserved phenomenon in Mixup training: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs, giving rise to a U-shaped generalization curve. This behavior is further aggravated when the size of original dataset is reduced. To help understand such a behavior of Mixup, we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data. Via analyzing a least-square regression problem with a random feature model, we explain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data. Extensive experiments are performed on a variety of benchmark datasets, validating this explanation.

翻译：混合训练（Mixup）是一种简单而有效的正则化技术，通过在随机样本对之间进行线性插值创建合成训练实例，从而提升使用SGD训练的深度模型的性能。在本工作中，我们报告了混合训练中一个先前未被观察到的现象：在多个标准数据集上，经过大量epoch的训练后，混合训练模型的性能开始下降，呈现出U形泛化曲线。当原始数据集规模减小时，这一行为会进一步加剧。为帮助理解混合训练的此类行为，我们从理论上证明，混合训练可能会向合成数据引入不期望的数据依赖标签噪声。通过分析随机特征模型下的最小二乘回归问题，我们解释了噪声标签为何可能导致U形曲线出现：混合训练在早期训练阶段通过拟合干净模式提升泛化能力，但随着训练进程，混合训练会过度拟合合成数据中的噪声。我们在多种基准数据集上进行了大量实验，验证了这一解释。