Despite the empirical success and practical significance of (relational) knowledge distillation that matches (the relations of) features between teacher and student models, the corresponding theoretical interpretations remain limited for various knowledge distillation paradigms. In this work, we take an initial step toward a theoretical understanding of relational knowledge distillation (RKD), with a focus on semi-supervised classification problems. We start by casting RKD as spectral clustering on a population-induced graph unveiled by a teacher model. Via a notion of clustering error that quantifies the discrepancy between the predicted and ground truth clusterings, we illustrate that RKD over the population provably leads to low clustering error. Moreover, we provide a sample complexity bound for RKD with limited unlabeled samples. For semi-supervised learning, we further demonstrate the label efficiency of RKD through a general framework of cluster-aware semi-supervised learning that assumes low clustering errors. Finally, by unifying data augmentation consistency regularization into this cluster-aware framework, we show that despite the common effect of learning accurate clusterings, RKD facilitates a "global" perspective through spectral clustering, whereas consistency regularization focuses on a "local" perspective via expansion.

翻译：摘要：尽管（关系）知识蒸馏在实证上取得了成功且具有实际重要性——这种方法能够匹配教师模型与学生模型之间的特征（关系），但针对各种知识蒸馏范式的理论解释仍十分有限。本文首次尝试从理论上理解关系知识蒸馏（RKD），重点聚焦于半监督分类问题。我们首先将RKD重新阐释为对由教师模型揭示的群体诱导图进行谱聚类。通过引入量化预测聚类与真实聚类之间差异的聚类误差概念，我们证明基于群体的RKD能够以可证明的方式实现低聚类误差。此外，我们给出了在有限未标注样本条件下RKD的样本复杂度界。针对半监督学习，我们进一步通过一个假设低聚类误差的通用聚类感知半监督学习框架，论证了RKD的标签效率。最后，通过将数据增强一致性正则化统一归入该聚类感知框架，我们证明了尽管两种方法都能有效学习精确聚类，但RKD通过谱聚类促进了一种"全局"视角，而一致性正则化则通过扩展机制专注于"局部"视角。