Diffusion models are a class of flexible generative models trained with an approximation to the log-likelihood objective. However, most use cases of diffusion models are not concerned with likelihoods, but instead with downstream objectives such as human-perceived image quality or drug effectiveness. In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for such objectives. We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms, which we refer to as denoising diffusion policy optimization (DDPO), that are more effective than alternative reward-weighted likelihood approaches. Empirically, DDPO is able to adapt text-to-image diffusion models to objectives that are difficult to express via prompting, such as image compressibility, and those derived from human feedback, such as aesthetic quality. Finally, we show that DDPO can improve prompt-image alignment using feedback from a vision-language model without the need for additional data collection or human annotation. The project's website can be found at http://rl-diffusion.github.io .

翻译：扩散模型是一类灵活的生成模型，通过近似对数似然目标进行训练。然而，扩散模型的大部分应用场景并不关心似然值本身，而是关注下游目标，例如人类感知的图像质量或药物有效性。在本文中，我们研究了直接针对此类目标优化扩散模型的强化学习方法。我们阐述了如何将去噪过程建模为多步决策问题，从而使得一类策略梯度算法（我们称之为去噪扩散策略优化，DDPO）能够比替代的奖励加权似然方法更有效。实验表明，DDPO能够使文本到图像的扩散模型适应难以通过提示表达的优化目标（如图像可压缩性）以及源自人类反馈的目标（如美学质量）。最后，我们证明DDPO可利用视觉-语言模型的反馈来提升提示与图像的对齐度，无需额外的数据收集或人工标注。项目网站见http://rl-diffusion.github.io。