Using reinforcement learning with human feedback (RLHF) has shown significant promise in fine-tuning diffusion models. Previous methods start by training a reward model that aligns with human preferences, then leverage RL techniques to fine-tune the underlying models. However, crafting an efficient reward model demands extensive datasets, optimal architecture, and manual hyperparameter tuning, making the process both time and cost-intensive. The direct preference optimization (DPO) method, effective in fine-tuning large language models, eliminates the necessity for a reward model. However, the extensive GPU memory requirement of the diffusion model's denoising process hinders the direct application of the DPO method. To address this issue, we introduce the Direct Preference for Denoising Diffusion Policy Optimization (D3PO) method to directly fine-tune diffusion models. The theoretical analysis demonstrates that although D3PO omits training a reward model, it effectively functions as the optimal reward model trained using human feedback data to guide the learning process. This approach requires no training of a reward model, proving to be more direct, cost-effective, and minimizing computational overhead. In experiments, our method uses the relative scale of objectives as a proxy for human preference, delivering comparable results to methods using ground-truth rewards. Moreover, D3PO demonstrates the ability to reduce image distortion rates and generate safer images, overcoming challenges lacking robust reward models. Our code is publicly available in https://github.com/yk7333/D3PO/tree/main.

翻译：使用基于人类反馈的强化学习（RLHF）在微调扩散模型方面展示了显著潜力。以往方法首先训练一个与人类偏好对齐的奖励模型，随后利用强化学习技术微调基础模型。然而，构建高效奖励模型需要大量数据集、最优架构和人工超参数调优，这一过程既耗时又成本高昂。直接偏好优化（DPO）方法在微调大语言模型中效果显著，可消除对奖励模型的需求。但扩散模型去噪过程对GPU内存的巨大需求阻碍了DPO方法的直接应用。为解决该问题，我们提出去噪扩散策略优化直接偏好（D3PO）方法，用于直接微调扩散模型。理论分析表明，尽管D3PO省略了奖励模型训练，但其能有效利用人类反馈数据指导学习过程，起到最优奖励模型的作用。该方法无需训练奖励模型，具有更直接、经济高效且计算开销更小的优势。在实验中，我们以目标相对尺度作为人类偏好代理指标，取得了与使用真实奖励方法相当的结果。此外，D3PO展现了降低图像失真率和生成更安全图像的能力，克服了缺乏稳健奖励模型的挑战。我们的代码已在https://github.com/yk7333/D3PO/tree/main公开。