Diffusion models have emerged as a key pillar of foundation models in visual domains. One of their critical applications is to universally solve different downstream inverse tasks via a single diffusion prior without re-training for each task. Most inverse tasks can be formulated as inferring a posterior distribution over data (e.g., a full image) given a measurement (e.g., a masked image). This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution. We show that our approach naturally leads to regularization by denoising diffusion process (RED-Diff) where denoisers at different timesteps concurrently impose different structural constraints over the image. To gauge the contribution of denoisers from different timesteps, we propose a weighting mechanism based on signal-to-noise-ratio (SNR). Our approach provides a new variational perspective for solving inverse problems with diffusion models, allowing us to formulate sampling as stochastic optimization, where one can simply apply off-the-shelf solvers with lightweight iterates. Our experiments for image restoration tasks such as inpainting and superresolution demonstrate the strengths of our method compared with state-of-the-art sampling-based diffusion models.

翻译：扩散模型已成为视觉领域基础模型的关键支柱。其核心应用之一是通过单一扩散先验，无需针对每项任务重新训练，即可通用地解决不同下游逆问题。大多数逆问题可表述为在给定测量值（例如带掩码图像）情况下，推断数据（例如完整图像）的后验分布。然而，这在扩散模型中具有挑战性，因为扩散过程的非线性和迭代特性使得后验分布难以处理。为应对这一挑战，我们提出了一种变分方法，该方法通过设计旨在逼近真实后验分布。我们证明该方法自然导出了去噪扩散过程正则化（RED-Diff），其中不同时间步的去噪器同时对图像施加不同的结构约束。为评估不同时间步去噪器的贡献，我们提出了一种基于信噪比（SNR）的加权机制。我们的方法为利用扩散模型求解逆问题提供了新的变分视角，使得我们可以将采样过程表述为随机优化，从而只需使用现成的求解器配合轻量级迭代即可。我们在图像修复任务（如图像修复和超分辨率）上的实验证明了我们的方法与最先进的基于采样的扩散模型相比具有优势。