In image restoration (IR), leveraging semantic priors from segmentation models has been a common approach to improve performance. The recent segment anything model (SAM) has emerged as a powerful tool for extracting advanced semantic priors to enhance IR tasks. However, the computational cost of SAM is prohibitive for IR, compared to existing smaller IR models. The incorporation of SAM for extracting semantic priors considerably hampers the model inference efficiency. To address this issue, we propose a general framework to distill SAM's semantic knowledge to boost exiting IR models without interfering with their inference process. Specifically, our proposed framework consists of the semantic priors fusion (SPF) scheme and the semantic priors distillation (SPD) scheme. SPF fuses two kinds of information between the restored image predicted by the original IR model and the semantic mask predicted by SAM for the refined restored image. SPD leverages a self-distillation manner to distill the fused semantic priors to boost the performance of original IR models. Additionally, we design a semantic-guided relation (SGR) module for SPD, which ensures semantic feature representation space consistency to fully distill the priors. We demonstrate the effectiveness of our framework across multiple IR models and tasks, including deraining, deblurring, and denoising.

翻译：在图像修复（IR）中，利用来自分割模型的语义先验知识已成为提升性能的常用方法。近年来，"分割一切模型"（SAM）作为提取高级语义先验的有力工具，能够增强图像修复任务的效果。然而，与现有规模较小的图像修复模型相比，SAM的运算成本高得令人望而却步。将SAM用于提取语义先验会严重降低模型推理效率。为解决该问题，我们提出了一种通用框架，旨在从SAM中提炼语义知识来增强现有图像修复模型，且不影响其推理过程。具体而言，所提框架包含语义先验融合（SPF）方案和语义先验蒸馏（SPD）方案。SPF将原始图像修复模型预测的修复图像与SAM预测的语义掩码中的两类信息进行融合，以生成更精细的修复图像。SPD则采用自蒸馏方式，将融合后的语义先验知识进行蒸馏，从而提升原始图像修复模型的性能。此外，我们为SPD设计了语义引导关系（SGR）模块，该模块通过确保语义特征表示空间的一致性来充分蒸馏先验知识。我们在多种图像修复模型和任务（包括去雨、去模糊及去噪）上验证了所提框架的有效性。