Direct Preference Optimization (DPO) has recently expanded its successful application from aligning large language models (LLMs) to aligning text-to-image models with human preferences, which has generated considerable interest within the community. However, we have observed that these approaches rely solely on minimizing the reverse Kullback-Leibler divergence during alignment process between the fine-tuned model and the reference model, neglecting the incorporation of other divergence constraints. In this study, we focus on extending reverse Kullback-Leibler divergence in the alignment paradigm of text-to-image models to $f$-divergence, which aims to garner better alignment performance as well as good generation diversity. We provide the generalized formula of the alignment paradigm under the $f$-divergence condition and thoroughly analyze the impact of different divergence constraints on alignment process from the perspective of gradient fields. We conduct comprehensive evaluation on image-text alignment performance, human value alignment performance and generation diversity performance under different divergence constraints, and the results indicate that alignment based on Jensen-Shannon divergence achieves the best trade-off among them. The option of divergence employed for aligning text-to-image models significantly impacts the trade-off between alignment performance (especially human value alignment) and generation diversity, which highlights the necessity of selecting an appropriate divergence for practical applications.

翻译：直接偏好优化（DPO）近期已将其成功应用从对齐大语言模型（LLM）扩展至对齐文本到图像模型与人类偏好，这在学界引起了广泛关注。然而，我们观察到现有方法在微调模型与参考模型的对齐过程中仅依赖于最小化反向Kullback-Leibler散度，未能纳入其他散度约束。本研究致力于将文本到图像模型对齐范式中的反向Kullback-Leibler散度扩展至$f$散度，旨在获得更优的对齐性能与良好的生成多样性。我们提出了$f$散度条件下对齐范式的广义公式，并从梯度场角度深入分析了不同散度约束对对齐过程的影响。通过对不同散度约束下的图文对齐性能、人类价值观对齐性能及生成多样性性能进行综合评估，结果表明基于Jensen-Shannon散度的对齐方法能实现三者间的最佳权衡。用于对齐文本到图像模型的散度选择显著影响着对齐性能（特别是人类价值观对齐）与生成多样性之间的平衡，这凸显了在实际应用中选择合适散度的必要性。