Self-distillation (SD) is the process of first training a \enquote{teacher} model and then using its predictions to train a \enquote{student} model with the \textit{same} architecture. Specifically, the student's objective function is $\big(\xi*\ell(\text{teacher's predictions}, \text{ student's predictions}) + (1-\xi)*\ell(\text{given labels}, \text{ student's predictions})\big)$, where $\ell$ is some loss function and $\xi$ is some parameter $\in [0,1]$. Empirically, SD has been observed to provide performance gains in several settings. In this paper, we theoretically characterize the effect of SD in two supervised learning problems with \textit{noisy labels}. We first analyze SD for regularized linear regression and show that in the high label noise regime, the optimal value of $\xi$ that minimizes the expected error in estimating the ground truth parameter is surprisingly greater than 1. Empirically, we show that $\xi > 1$ works better than $\xi \leq 1$ even with the cross-entropy loss for several classification datasets when 50\% or 30\% of the labels are corrupted. Further, we quantify when optimal SD is better than optimal regularization. Next, we analyze SD in the case of logistic regression for binary classification with random label corruption and quantify the range of label corruption in which the student outperforms the teacher in terms of accuracy. To our knowledge, this is the first result of its kind for the cross-entropy loss.

翻译：自蒸馏（SD）是指首先训练一个"教师"模型，然后利用其预测结果训练具有相同架构的"学生"模型的过程。具体而言，学生模型的目标函数为 $\big(\xi*\ell(\text{教师预测}, \text{学生预测}) + (1-\xi)*\ell(\text{给定标签}, \text{学生预测})\big)$，其中 $\ell$ 为损失函数，$\xi$ 为参数且 $\in [0,1]$。实验表明自蒸馏在多种场景下能带来性能提升。本文从理论层面刻画了自蒸馏在两类含噪声标签的有监督学习问题中的影响。首先针对正则化线性回归进行分析，发现在高标签噪声区间，使真实参数估计期望误差最小的最优 $\xi$ 值竟大于1。实验表明，当30%或50%的标签被污染时，采用交叉熵损失的多个分类数据集上 $\xi > 1$ 的效果优于 $\xi \leq 1$。进一步我们量化了最优自蒸馏优于最优正则化的条件。随后分析随机标签腐蚀场景下二分类逻辑回归中的自蒸馏，量化了学生模型在准确率上超越教师模型的标签腐蚀范围。据我们所知，这是关于交叉熵损失的首个此类理论结果。