This work aims to improve the applicability of diffusion models in realistic image restoration. Specifically, we enhance the diffusion model in several aspects such as network architecture, noise level, denoising steps, training image size, and optimizer/scheduler. We show that tuning these hyperparameters allows us to achieve better performance on both distortion and perceptual scores. We also propose a U-Net based latent diffusion model which performs diffusion in a low-resolution latent space while preserving high-resolution information from the original input for the decoding process. Compared to the previous latent-diffusion model which trains a VAE-GAN to compress the image, our proposed U-Net compression strategy is significantly more stable and can recover highly accurate images without relying on adversarial optimization. Importantly, these modifications allow us to apply diffusion models to various image restoration tasks, including real-world shadow removal, HR non-homogeneous dehazing, stereo super-resolution, and bokeh effect transformation. By simply replacing the datasets and slightly changing the noise network, our model, named Refusion, is able to deal with large-size images (e.g., 6000 x 4000 x 3 in HR dehazing) and produces good results on all the above restoration problems. Our Refusion achieves the best perceptual performance in the NTIRE 2023 Image Shadow Removal Challenge and wins 2nd place overall.

翻译：本文旨在提升扩散模型在真实图像复原任务中的适用性。具体而言，我们从网络架构、噪声水平、去噪步数、训练图像尺寸以及优化器/调度器等多个方面对扩散模型进行了改进。研究表明，调整这些超参数能够在失真度和感知质量评分上同时取得更优性能。我们提出一种基于U-Net的潜扩散模型，该模型在低分辨率潜空间执行扩散过程，同时保留原始输入的高分辨率信息用于解码阶段。相较于此前采用VAE-GAN进行图像压缩的潜扩散模型，本文提出的U-Net压缩策略具有显著更优的稳定性，可在不依赖对抗优化的情况下恢复高精度图像。关键的是，这些改进使扩散模型能够应用于多种图像复原任务，包括真实场景阴影去除、高分辨率非均匀去雾、立体超分辨率以及散景效果转换。通过简单替换数据集并微调噪声网络，本模型（命名为Refusion）可处理大尺寸图像（例如高分辨率去雾中的6000×4000×3），并在上述所有复原问题上取得优异效果。我们的Refusion在NTIRE 2023图像阴影去除挑战赛中获得最佳感知性能，并取得总排名第二。