The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.

翻译：从文本到图像用户中收集大规模人类偏好数据集的能力通常仅限于公司层面，导致此类数据集难以公开获取。为解决这一问题，我们开发了一个网络应用，使文本到图像用户能够生成图像并标注自身偏好。通过该应用，我们构建了Pick-a-Pic——一个包含文本到图像提示词及真实用户对生成图像偏好的大规模开放数据集。我们利用该数据集训练了基于CLIP的评分函数PickScore，其在预测人类偏好任务上展现出超越人类的性能。随后，我们测试了PickScore在模型评估中的能力，发现其与人类排序的相关性优于其他自动评估指标。因此，我们建议使用PickScore评估未来文本到图像生成模型，并采用Pick-a-Pic提示词作为比MS-COCO更具相关性的数据集。最后，我们展示了PickScore如何通过排序机制增强现有文本到图像模型。