The performance of a model trained with \textit{noisy labels} is often improved by simply \textit{retraining} the model with its own predicted \textit{hard} labels (i.e., $1$/$0$ labels). Yet, a detailed theoretical characterization of this phenomenon is lacking. In this paper, we theoretically analyze retraining in a linearly separable setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels. To the best of our knowledge, this is the first such theoretical result. Retraining finds application in improving training with label differential privacy (DP) which involves training with noisy labels. We empirically show that retraining selectively on the samples for which the predicted label matches the given label significantly improves label DP training at \textit{no extra privacy cost}; we call this \textit{consensus-based retraining}. For e.g., when training ResNet-18 on CIFAR-100 with $\epsilon=3$ label DP, we obtain $6.4\%$ improvement in accuracy with consensus-based retraining.

翻译：在\textit{噪声标签}下训练的模型，若使用其自身预测的\textit{硬}标签（即$1$/$0$标签）进行简单\textit{再训练}，性能常能得到提升。然而，目前尚缺乏对这一现象详细的理论刻画。本文在线性可分设定下，对给定随机扰动标签的再训练过程进行理论分析，证明再训练能够提升通过初始（带噪）标签训练获得的总体准确率。据我们所知，这是首个此类理论结果。再训练可应用于改进带标签差分隐私（DP）的训练过程，该过程本就涉及噪声标签下的训练。我们通过实验表明，仅对预测标签与给定标签一致的样本进行选择性再训练，能以\textit{零额外隐私代价}显著提升标签DP训练效果；我们称此方法为\textit{基于共识的再训练}。例如，在CIFAR-100数据集上以$\epsilon=3$的标签DP训练ResNet-18时，采用基于共识的再训练可获得$6.4\%$的准确率提升。