Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create VisionPrefer, a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct VisionPrefer. To validate the effectiveness of VisionPrefer, we train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators. Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of VisionPrefer, and extensive experimental results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover, VisionPrefer indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.

翻译：近期研究已证明，利用人类偏好数据集优化文本生成图像模型具有显著潜力，能增强生成图像与文本提示的对齐程度。然而，现有的人类偏好数据集要么构建成本过高，要么在偏好维度上缺乏多样性，导致其在开源文本生成图像模型的指令微调中适用性有限，阻碍了进一步探索。为解决这些挑战并推动通过指令微调实现生成模型的对齐，我们利用多模态大语言模型创建了VisionPrefer——一个高质量、细粒度、涵盖多维度偏好特征的数据集。我们汇聚来自AI标注器在提示遵循、美学、保真度和无害性四个维度的反馈，构建了VisionPrefer。为验证VisionPrefer的有效性，我们在其上训练了奖励模型VP-Score，用于指导文本生成图像模型的训练，其偏好预测精度可与人类标注者媲美。此外，我们采用两种强化学习方法对生成模型进行监督微调以评估VisionPrefer的性能，大量实验结果表明，VisionPrefer在组合图像生成中显著提升了文本-图像对齐水平（尤其在美学等维度），并在多种图像分布上的泛化能力优于先前的人类偏好度量指标。更关键的是，VisionPrefer表明，将AI生成的合成数据作为监督信号，是实现视觉生成模型与人类偏好更优对齐的有前景途径。