The advancement of knowledge distillation has played a crucial role in enabling the transfer of knowledge from larger teacher models to smaller and more efficient student models, and is particularly beneficial for online and resource-constrained applications. The effectiveness of the student model heavily relies on the quality of the distilled knowledge received from the teacher. Given the accessibility of unlabelled remote sensing data, semi-supervised learning has become a prevalent strategy for enhancing model performance. However, relying solely on semi-supervised learning with smaller models may be insufficient due to their limited capacity for feature extraction. This limitation restricts their ability to exploit training data. To address this issue, we propose an integrated approach that combines knowledge distillation and semi-supervised learning methods. This hybrid approach leverages the robust capabilities of large models to effectively utilise large unlabelled data whilst subsequently providing the small student model with rich and informative features for enhancement. The proposed semi-supervised learning-based knowledge distillation (SSLKD) approach demonstrates a notable improvement in the performance of the student model, in the application of road segmentation, surpassing the effectiveness of traditional semi-supervised learning methods.

翻译：知识蒸馏技术的进步在实现从大型教师模型向更小、更高效的学生模型传递知识方面发挥了关键作用，尤其有利于在线和资源受限的应用场景。学生模型的有效性高度依赖于从教师模型处获得的蒸馏知识质量。鉴于无标注遥感数据的可获取性，半监督学习已成为提升模型性能的主流策略。然而，单纯依赖小模型的半监督学习可能因其有限的特征提取能力而效果不足，这种局限性制约了模型对训练数据的利用效率。为解决此问题，我们提出一种融合知识蒸馏与半监督学习的综合方法。这种混合方法充分利用大型模型的强大能力来有效处理大规模无标注数据，同时为学生小模型提供丰富且具信息量的特征以增强其性能。所提出的基于半监督学习的知识蒸馏（SSLKD）方法在道路分割应用中显著提升了学生模型的性能，其效果超越了传统半监督学习方法。