In image restoration (IR), leveraging semantic priors from segmentation models has been a common approach to improve performance. The recent segment anything model (SAM) has emerged as a powerful tool for extracting advanced semantic priors to enhance IR tasks. However, the computational cost of SAM is prohibitive for IR, compared to existing smaller IR models. The incorporation of SAM for extracting semantic priors considerably hampers the model inference efficiency. To address this issue, we propose a general framework to distill SAM's semantic knowledge to boost exiting IR models without interfering with their inference process. Specifically, our proposed framework consists of the semantic priors fusion (SPF) scheme and the semantic priors distillation (SPD) scheme. SPF fuses two kinds of information between the restored image predicted by the original IR model and the semantic mask predicted by SAM for the refined restored image. SPD leverages a self-distillation manner to distill the fused semantic priors to boost the performance of original IR models. Additionally, we design a semantic-guided relation (SGR) module for SPD, which ensures semantic feature representation space consistency to fully distill the priors. We demonstrate the effectiveness of our framework across multiple IR models and tasks, including deraining, deblurring, and denoising.

翻译：在图像恢复（IR）中，利用分割模型的语义先验是提升性能的常用方法。近期出现的分割一切模型（SAM）已成为提取高级语义先验以增强IR任务的强大工具。然而，与现有的小型IR模型相比，SAM的计算成本过高，难以直接用于IR。引入SAM提取语义先验会显著降低模型推理效率。为解决这一问题，本文提出一个通用框架，在不干扰现有IR模型推理过程的前提下，蒸馏SAM的语义知识以提升其性能。具体而言，该框架包含语义先验融合（SPF）方案和语义先验蒸馏（SPD）方案：SPF融合原始IR模型预测的恢复图像与SAM预测的语义掩码中的两类信息，生成精细化恢复图像；SPD通过自蒸馏方式将融合后的语义先验蒸馏到原始IR模型中，以提升其性能。此外，我们针对SPD设计了语义引导关系（SGR）模块，通过确保语义特征表示空间的一致性来充分蒸馏先验知识。我们在去雨、去模糊和去噪等多个IR模型和任务上验证了该框架的有效性。