The core of knowledge distillation lies in transferring the teacher's rich 'dark knowledge'-subtle probabilistic patterns that reveal how classes are related and the distribution of uncertainties. While this idea is well established, teachers trained with conventional cross-entropy often fail to preserve such signals. Their distributions collapse into sharp, overconfident peaks that appear decisive but are in fact brittle, offering little beyond the hard label or subtly hindering representation-level transfer. This overconfidence is especially problematic in high-cardinality tasks, where the nuances among many plausible classes matter most for guiding a compact student. Moreover, such brittle targets reduce robustness under distribution shift, leaving students vulnerable to miscalibration in real-world conditions. To address this limitation, we revisit distillation from a distributional perspective and propose Calibrated Uncertainty Distillation (CUD), a framework designed to make dark knowledge more faithfully accessible. Instead of uncritically adopting the teacher's overconfidence, CUD encourages teachers to reveal uncertainty where it is informative and guides students to learn from targets that are calibrated rather than sharpened certainty. By directly shaping the teacher's predictive distribution before transfer, our approach balances accuracy and calibration, allowing students to benefit from both confident signals on easy cases and structured uncertainty on hard ones. Across diverse benchmarks, CUD yields students that are not only more accurate, but also more calibrated under shift and more reliable on ambiguous, long-tail inputs.

翻译：知识蒸馏的核心在于传递教师模型丰富的"暗知识"——揭示类别间关联性与不确定性分布的细微概率模式。尽管这一理念已广为接受，但使用传统交叉熵训练的教师模型往往无法保留此类信号。其预测分布会坍缩为尖锐的过置信峰值，这些峰值看似确定实则脆弱，除硬标签外几乎无法提供额外信息，甚至可能微妙地阻碍表示层面的迁移。这种过置信现象在高基数任务中尤为严重，因为众多可能类别间的细微差异对于指导紧凑的学生模型至关重要。此外，此类脆弱目标会降低分布偏移下的鲁棒性，使学生模型在实际场景中容易产生校准误差。为解决这一局限，我们从分布视角重新审视蒸馏过程，提出校准不确定性蒸馏框架，旨在更忠实地提取暗知识。该方法并非盲目接受教师的过置信预测，而是鼓励教师在有信息量的区域展现不确定性，并引导学生从经过校准而非过度锐化的目标中学习。通过在知识迁移前直接调整教师的预测分布，我们的方法在准确性与校准性之间取得平衡，使学生既能从简单样本的置信信号中获益，也能从困难样本的结构化不确定性中学习。在多样化基准测试中，校准不确定性蒸馏框架训练出的学生模型不仅具有更高精度，而且在分布偏移下表现出更好的校准特性，对模糊长尾输入也具有更强的可靠性。