Knowledge Distillation (KD) is one proposed solution to large model sizes and slow inference speed in semantic segmentation. In our research we identify 25 proposed distillation loss terms from 14 publications in the last 4 years. Unfortunately, a comparison of terms based on published results is often impossible, because of differences in training configurations. A good illustration of this problem is the comparison of two publications from 2022. Using the same models and dataset, Structural and Statistical Texture Distillation (SSTKD) reports an increase of student mIoU of 4.54 and a final performance of 29.19, while Adaptive Perspective Distillation (APD) only improves student performance by 2.06 percentage points, but achieves a final performance of 39.25. The reason for such extreme differences is often a suboptimal choice of hyperparameters and a resulting underperformance of the student model used as reference point. In our work, we reveal problems of insufficient hyperparameter tuning by showing that distillation improvements of two widely accepted frameworks, SKD and IFVD, vanish when hyperparameters are optimized sufficiently. To improve comparability of future research in the field, we establish a solid baseline for three datasets and two student models and provide extensive information on hyperparameter tuning. We find that only two out of eight techniques can compete with our simple baseline on the ADE20K dataset.

翻译：知识蒸馏（KD）是解决语义分割中模型规模大、推理速度慢问题的一种方案。在我们的研究中，我们从近4年的14篇论文中识别出25个提出的蒸馏损失项。然而，由于训练配置的差异，基于已发表结果对这些损失项进行比较往往是不可能的。这一问题的一个典型例证是2022年两篇论文的比较：使用相同的模型和数据集，结构及统计纹理蒸馏（SSTKD）报告学生模型的平均交并比（mIoU）提升了4.54，最终性能达到29.19；而自适应视角蒸馏（APD）仅将学生模型性能提升了2.06个百分点，最终性能却达到39.25。造成如此巨大差异的原因通常是超参数选择的次优性，以及作为参考点的学生模型性能欠佳。在我们的工作中，通过揭示两个广泛接受的框架SKD和IFVD在充分优化超参数后其蒸馏改进效果消失的现象，我们指出了超参数调优不足的问题。为提升该领域未来研究的可比性，我们针对三个数据集和两个学生模型建立了稳健的基线，并提供了超参数调优的详细信息。我们发现，在ADE20K数据集上，八种技术中仅有两种能与我们的简单基线相媲美。