Knowledge Distillation (KD) is a central paradigm for transferring knowledge from a large teacher network to a typically smaller student model, often by leveraging soft probabilistic outputs. While KD has shown strong empirical success in numerous applications, its theoretical underpinnings remain only partially understood. In this work, we adopt a Bayesian perspective on KD to rigorously analyze the convergence behavior of students trained with Stochastic Gradient Descent (SGD). We study two regimes: $(i)$ when the teacher provides the exact Bayes Class Probabilities (BCPs); and $(ii)$ supervision with noisy approximations of the BCPs. Our analysis shows that learning from BCPs yields variance reduction and removes neighborhood terms in the convergence bounds compared to one-hot supervision. We further characterize how the level of noise affects generalization and accuracy. Motivated by these insights, we advocate the use of Bayesian deep learning models, which typically provide improved estimates of the BCPs, as teachers in KD. Consistent with our analysis, we experimentally demonstrate that students distilled from Bayesian teachers not only achieve higher accuracies (up to +4.27%), but also exhibit more stable convergence (up to 30% less noise), compared to students distilled from deterministic teachers.

翻译：知识蒸馏（KD）是通过利用软概率输出来将知识从大型教师网络迁移到通常较小的学生模型的核心范式。尽管KD在众多应用中展现出强大的实证效果，但其理论基础仍仅得到部分理解。在本工作中，我们采用贝叶斯视角对KD进行严格分析，以研究使用随机梯度下降（SGD）训练的学生模型的收敛行为。我们研究了两种机制：$(i)$ 当教师提供精确的贝叶斯分类概率（BCPs）时；以及$(ii)$ 使用BCPs的噪声近似进行监督。我们的分析表明，与独热编码监督相比，从BCPs中学习能实现方差缩减，并在收敛界中消除邻域项。我们进一步刻画了噪声水平如何影响泛化能力和准确性。基于这些见解，我们提倡在KD中使用贝叶斯深度学习模型作为教师，这类模型通常能提供改进的BCPs估计。与我们的分析一致，实验证明，与从确定性教师蒸馏的学生相比，从贝叶斯教师蒸馏的学生不仅实现了更高的准确率（最高提升+4.27%），而且表现出更稳定的收敛性（噪声减少高达30%）。