Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of a time-varying sequence of log-probability density functions. This interpretation has motivated classifier-based and classifier-free guidance as methods for post-hoc control of diffusion models. In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance. In particular, we investigate why certain types of composition fail using current techniques and present a number of solutions. We conclude that the sampler (not the model) is responsible for this failure and propose new samplers, inspired by MCMC, which enable successful compositional generation. Further, we propose an energy-based parameterization of diffusion models which enables the use of new compositional operators and more sophisticated, Metropolis-corrected samplers. Intriguingly we find these samplers lead to notable improvements in compositional generation across a wide set of problems such as classifier-guided ImageNet modeling and compositional text-to-image generation.

翻译：自提出以来，扩散模型已迅速成为许多领域中生成式建模的主流方法。它们可被解释为学习随时间变化的对数概率密度函数序列的梯度。这一解释催生了基于分类器和无分类器的引导方法，用于扩散模型的事后控制。本文基于扩散模型的分数解释，探索对扩散模型进行条件化、修改及复用的替代方案，以处理涉及组成式生成与引导的任务。具体而言，我们研究了当前技术下某些类型组合失败的原因，并提出一系列解决方案。我们得出结论：采样器（而非模型）是导致失败的原因，并受MCMC启发提出了新型采样器，从而实现成功的组成式生成。此外，我们提出了一种基于能量的扩散模型参数化方法，该方法支持新组合运算符及更复杂的、经Metropolis校正的采样器的使用。有趣的是，我们发现这些采样器能在广泛的问题（如基于分类器引导的ImageNet建模与组成式文本到图像生成）中显著提升组成式生成效果。