Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network. However, different from image synthesis generating each pixel from scratch, most pixels of image restoration (IR) are given. Thus, for IR, traditional DMs running massive iterations on a large model to estimate whole images or feature maps is inefficient. To address this issue, we propose an efficient DM for IR (DiffIR), which consists of a compact IR prior extraction network (CPEN), dynamic IR transformer (DIRformer), and denoising network. Specifically, DiffIR has two training stages: pretraining and training DM. In pretraining, we input ground-truth images into CPEN$_{S1}$ to capture a compact IR prior representation (IPR) to guide DIRformer. In the second stage, we train the DM to directly estimate the same IRP as pretrained CPEN$_{S1}$ only using LQ images. We observe that since the IPR is only a compact vector, DiffIR can use fewer iterations than traditional DM to obtain accurate estimations and generate more stable and realistic results. Since the iterations are few, our DiffIR can adopt a joint optimization of CPEN$_{S2}$, DIRformer, and denoising network, which can further reduce the estimation error influence. We conduct extensive experiments on several IR tasks and achieve SOTA performance while consuming less computational costs.

翻译：扩散模型（DM）通过将图像生成过程建模为去噪网络的顺序应用，已取得最先进的性能。然而，与从零生成每个像素的图像合成不同，图像复原任务中大部分像素是已知的。因此，传统DM需在大型模型上运行大量迭代来估计整幅图像或特征图，这对图像复原而言效率低下。为解决此问题，我们提出一种面向图像复原的高效扩散模型（DiffIR），其由紧凑型图像复原先验提取网络（CPEN）、动态图像复原Transformer（DIRformer）和去噪网络组成。具体而言，DiffIR包含两个训练阶段：预训练和扩散模型训练。在预训练阶段，我们将真实图像输入CPEN$_{S1}$以捕获紧凑的图像复原先验表征（IPR），从而指导DIRformer。在第二阶段，我们仅使用低质量图像训练DM，使其直接估计与预训练CPEN$_{S1}相同的IPR。我们观察到，由于IPR仅为紧凑向量，DiffIR可通过比传统DM更少的迭代次数获得精确估计，并生成更稳定逼真的结果。因迭代次数较少，我们的DiffIR能对CPEN$_{S2}$、DIRformer和去噪网络进行联合优化，从而进一步降低估计误差的影响。我们在多个图像复原任务上开展大量实验，在降低计算成本的同时取得了最先进的性能。