Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of a time-varying sequence of log-probability density functions. This interpretation has motivated classifier-based and classifier-free guidance as methods for post-hoc control of diffusion models. In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance. In particular, we investigate why certain types of composition fail using current techniques and present a number of solutions. We conclude that the sampler (not the model) is responsible for this failure and propose new samplers, inspired by MCMC, which enable successful compositional generation. Further, we propose an energy-based parameterization of diffusion models which enables the use of new compositional operators and more sophisticated, Metropolis-corrected samplers. Intriguingly we find these samplers lead to notable improvements in compositional generation across a wide set of problems such as classifier-guided ImageNet modeling and compositional text-to-image generation.

翻译：自扩散模型问世以来，其已迅速成为众多领域生成式建模的主流方法。这类模型可被解释为学习时变对数概率密度函数序列的梯度。这一解释催生了基于分类器和无分类器引导方法，用于对扩散模型进行事后控制。本研究基于扩散模型的得分解释，探索以条件化、修改和复用扩散模型完成复合生成与引导任务的新路径。具体而言，我们分析了现有技术导致某些复合生成失败的原因，并提出若干解决方案。研究证实采样器（而非模型）是造成该缺陷的根本因素，据此提出受MCMC启发的新型采样器，成功实现了复合生成。此外，我们提出一种基于能量的扩散模型参数化方法，使得新型复合算子与更复杂的梅特罗波利斯校正采样器得以应用。值得注意的是，这些采样器在多个问题上显著提升了复合生成性能，包括分类器引导的ImageNet建模与复合文本到图像生成。