Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of a time-varying sequence of log-probability density functions. This interpretation has motivated classifier-based and classifier-free guidance as methods for post-hoc control of diffusion models. In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance. In particular, we investigate why certain types of composition fail using current techniques and present a number of solutions. We conclude that the sampler (not the model) is responsible for this failure and propose new samplers, inspired by MCMC, which enable successful compositional generation. Further, we propose an energy-based parameterization of diffusion models which enables the use of new compositional operators and more sophisticated, Metropolis-corrected samplers. Intriguingly we find these samplers lead to notable improvements in compositional generation across a wide set of problems such as classifier-guided ImageNet modeling and compositional text-to-image generation.

翻译：自引入以来，扩散模型已迅速成为许多领域生成建模的主流方法。这些模型可被解释为学习随时间变化的对数概率密度函数序列的梯度。这一解释推动了基于分类器和无分类器的引导方法，用于对扩散模型进行事后控制。在本工作中，我们基于扩散模型的得分解释进一步发展这些思想，探索在涉及复合生成与引导任务中条件化、修改及复用扩散模型的替代方案。具体而言，我们研究了当前技术导致特定类型复合失败的原因，并提出若干解决途径。我们得出结论：采样器（而非模型）是造成该失败的主因，并受MCMC启发提出了能够实现成功复合生成的新型采样器。此外，我们提出一种基于能量的扩散模型参数化方法，支持使用新型复合算子及更复杂的梅特罗波利斯校正采样器。引人注目的是，这些采样器在分类器引导的ImageNet建模和复合文本到图像生成等广泛问题中，显著提升了复合生成性能。