Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs). While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing a certain volume of data. Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images ("winner" and "loser" images) for each text prompt. In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the diffusion model engages in competition with its earlier versions, facilitating an iterative self-improvement process. Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment. Our experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms the existing supervised fine-tuning method in aspects of human preference alignment and visual appeal right from its first iteration. By the second iteration, it exceeds the performance of RLHF-based methods across all metrics, achieving these results with less data.

翻译：微调扩散模型在生成式人工智能（GenAI）领域仍是一个尚未充分探索的前沿方向，尤其是与大型语言模型（LLMs）微调取得的显著进展相比。尽管Stable Diffusion（SD）和SDXL等前沿扩散模型依赖监督微调，但它们的性能在接触到一定量数据后不可避免地陷入瓶颈。近来，强化学习（RL）被用于通过人类偏好数据微调扩散模型，但这要求每个文本提示至少需要两张图像（“优胜者”和“落败者”图像）。本文提出一种创新技术——扩散模型自博弈微调（SPIN-Diffusion），其中扩散模型与其早期版本进行竞争，从而促进迭代式的自我优化过程。我们的方法为传统的监督微调和RL策略提供了替代方案，显著提升了模型性能与对齐度。在Pick-a-Pic数据集上的实验表明，SPIN-Diffusion从首次迭代起，就在人类偏好对齐和视觉吸引力方面超越了现有的监督微调方法。到第二次迭代时，其所有指标均优于基于RLHF的方法，且使用了更少的数据就能达到这些结果。