Recently, Direct Preference Optimization (DPO) has extended its success from aligning large language models (LLMs) to aligning text-to-image diffusion models with human preferences. Unlike most existing DPO methods that assume all diffusion steps share a consistent preference order with the final generated images, we argue that this assumption neglects step-specific denoising performance and that preference labels should be tailored to each step's contribution. To address this limitation, we propose Step-aware Preference Optimization (SPO), a novel post-training approach that independently evaluates and adjusts the denoising performance at each step, using a step-aware preference model and a step-wise resampler to ensure accurate step-aware supervision. Specifically, at each denoising step, we sample a pool of images, find a suitable win-lose pair, and, most importantly, randomly select a single image from the pool to initialize the next denoising step. This step-wise resampler process ensures the next win-lose image pair comes from the same image, making the win-lose comparison independent of the previous step. To assess the preferences at each step, we train a separate step-aware preference model that can be applied to both noisy and clean images. Our experiments with Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms the latest Diffusion-DPO in aligning generated images with complex, detailed prompts and enhancing aesthetics, while also achieving more than 20x times faster in training efficiency. Code and model: https://rockeycoss.github.io/spo.github.io/

翻译：近期，直接偏好优化（DPO）已将其成功从对齐大型语言模型（LLMs）扩展到对齐文本到图像扩散模型与人类偏好。与现有大多数DPO方法假设所有扩散步骤与最终生成图像共享一致偏好顺序不同，我们认为这一假设忽略了步特定的去噪性能，且偏好标签应针对每一步的贡献进行定制。为解决此局限，我们提出步感知偏好优化（SPO），这是一种新颖的后训练方法，通过步感知偏好模型和步级重采样器独立评估并调整每一步的去噪性能，确保准确的步感知监督。具体而言，在每一步去噪中，我们采样一组图像，找到合适的胜负对，并且最关键的是，从该组中随机选择一张图像以初始化下一步去噪。此步级重采样过程确保下一步的胜负图像对来自同一张图像，使得胜负比较独立于前一步。为评估每一步的偏好，我们训练了一个独立的步感知偏好模型，该模型可同时应用于含噪和干净图像。我们使用Stable Diffusion v1.5和SDXL进行的实验表明，SPO在将生成图像与复杂、详细提示对齐以及增强美观性方面显著优于最新的Diffusion-DPO，同时训练效率提升超过20倍。代码与模型：https://rockeycoss.github.io/spo.github.io/