Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of a time-varying sequence of log-probability density functions. This interpretation has motivated classifier-based and classifier-free guidance as methods for post-hoc control of diffusion models. In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance. In particular, we investigate why certain types of composition fail using current techniques and present a number of solutions. We conclude that the sampler (not the model) is responsible for this failure and propose new samplers, inspired by MCMC, which enable successful compositional generation. Further, we propose an energy-based parameterization of diffusion models which enables the use of new compositional operators and more sophisticated, Metropolis-corrected samplers. Intriguingly we find these samplers lead to notable improvements in compositional generation across a wide set of problems such as classifier-guided ImageNet modeling and compositional text-to-image generation.

翻译：自提出以来，扩散模型已迅速成为许多领域生成建模的主流方法。这类模型可被解释为学习时变对数概率密度函数序列的梯度，这一理解推动了基于分类器与无分类器的引导方法，用于对扩散模型进行事后控制。本研究基于扩散模型的分数解释框架，深入探讨了如何对扩散模型进行条件控制、修改及复用以完成涉及合成式生成与引导的任务。我们特别研究了现有技术在特定类型合成任务中失败的原因，并提出若干解决方案。我们发现采样器（而非模型）是导致该失败的关键因素，受MCMC启发提出了新型采样器，成功实现了合成式生成。进一步地，我们提出了扩散模型的能量参数化方法，该方法支持新型合成算子及更复杂的梅特罗波利斯校正采样器。有趣的是，这些采样器在分类器引导的ImageNet建模和合成式文本到图像生成等广泛任务中，显著提升了合成式生成的效果。