Knowledge distillation (KD) emerges as a challenging yet promising technique for compressing deep learning models, characterized by the transmission of extensive learning representations from proficient and computationally intensive teacher models to compact student models. However, only a handful of studies have endeavored to compress the models for single image super-resolution (SISR) through KD, with their effects on student model enhancement remaining marginal. In this paper, we put forth an approach from the perspective of efficient data utilization, namely, the Data Upcycling Knowledge Distillation (DUKD) which facilitates the student model by the prior knowledge teacher provided via upcycled in-domain data derived from their inputs. This upcycling process is realized through two efficient image zooming operations and invertible data augmentations which introduce the label consistency regularization to the field of KD for SISR and substantially boosts student model's generalization. The DUKD, due to its versatility, can be applied across a broad spectrum of teacher-student architectures. Comprehensive experiments across diverse benchmarks demonstrate that our proposed DUKD method significantly outperforms previous art, exemplified by an increase of up to 0.5dB in PSNR over baselines methods, and a 67% parameters reduced RCAN model's performance remaining on par with that of the RCAN teacher model.

翻译：知识蒸馏（KD）作为一种压缩深度学习模型的技术，具有挑战性且前景广阔，其特点是将高效且计算密集的教师模型中的广泛学习表征传递给紧凑的学生模型。然而，仅有少数研究尝试通过KD压缩单幅图像超分辨率（SISR）模型，且对学生模型性能的提升效果仍然有限。本文从高效数据利用的角度提出一种方法，即数据回用知识蒸馏（DUKD），该方法通过教师模型利用来自输入数据的域内数据回用所提供的先验知识，来促进学生模型的学习。这一回用过程通过两种高效的图像缩放操作和可逆数据增强实现，将标签一致性正则化引入SISR领域的KD中，并显著提升了学生模型的泛化能力。由于DUKD的通用性，其可广泛应用于各种教师-学生架构。在多个基准测试上的全面实验表明，我们提出的DUKD方法显著优于以往技术，例如在PSNR上比基线方法提升高达0.5dB，并且参数减少67%的RCAN模型的性能与RCAN教师模型相当。