Denoising diffusion models are a powerful type of generative models used to capture complex distributions of real-world signals. However, their applicability is limited to scenarios where training samples are readily available, which is not always the case in real-world applications. For example, in inverse graphics, the goal is to generate samples from a distribution of 3D scenes that align with a given image, but ground-truth 3D scenes are unavailable and only 2D images are accessible. To address this limitation, we propose a novel class of denoising diffusion probabilistic models that learn to sample from distributions of signals that are never directly observed. Instead, these signals are measured indirectly through a known differentiable forward model, which produces partial observations of the unknown signal. Our approach involves integrating the forward model directly into the denoising process. This integration effectively connects the generative modeling of observations with the generative modeling of the underlying signals, allowing for end-to-end training of a conditional generative model over signals. During inference, our approach enables sampling from the distribution of underlying signals that are consistent with a given partial observation. We demonstrate the effectiveness of our method on three challenging computer vision tasks. For instance, in the context of inverse graphics, our model enables direct sampling from the distribution of 3D scenes that align with a single 2D input image.

翻译：去噪扩散模型是一类强大的生成模型，用于捕捉真实世界信号的复杂分布。然而，其适用性仅限于训练样本可直接获取的场景，而这在实际应用中并不总是成立。例如，在逆图形学中，目标是生成与给定图像一致的3D场景分布样本，但真实3D场景不可获取，仅能获得2D图像。为解决这一局限，我们提出了一类新型去噪扩散概率模型，该模型学习从从未被直接观测到的信号分布中采样。这些信号通过已知的可微前向模型间接测量，该模型生成未知信号的部分观测结果。我们的方法将前向模型直接整合到去噪过程中。这种整合有效地将观测值的生成建模与底层信号的生成建模联系起来，从而实现对信号的条件生成模型的端到端训练。在推理阶段，我们的方法能够从与给定部分观测一致的底层信号分布中采样。我们在三个具有挑战性的计算机视觉任务上展示了该方法的有效性。例如，在逆图形学背景下，我们的模型能够直接从与单个2D输入图像一致的3D场景分布中采样。