Text-to-Image (T2I) generative models are becoming more crucial in terms of their ability to generate complex and high-quality images, which also raises concerns about the social biases in their outputs, especially in human generation. Sociological research has established systematic classifications of bias; however, existing research of T2I models often conflates different types of bias, hindering the progress of these methods. In this paper, we introduce BIGbench, a unified benchmark for Biases of Image Generation with a well-designed dataset. In contrast to existing benchmarks, BIGbench classifies and evaluates complex biases into four dimensions: manifestation of bias, visibility of bias, acquired attributes, and protected attributes. Additionally, BIGbench applies advanced multi-modal large language models (MLLM), achieving fully automated evaluation while maintaining high accuracy. We apply BIGbench to evaluate eight recent general T2I models and three debiased methods. We also conduct human evaluation, whose results demonstrated the effectiveness of BIGbench in aligning images and identifying various biases. Besides, our study also revealed new research directions about biases, including the side-effect of irrelevant protected attributes and distillation. Our dataset and benchmark is openly accessible to the research community to ensure the reproducibility.

翻译：文本到图像（T2I）生成模型因其生成复杂且高质量图像的能力而日益重要，这也引发了对其输出中社会偏见的担忧，尤其是在人物生成方面。社会学研究已建立系统性的偏见分类；然而，现有关于T2I模型的研究常常混淆不同类型的偏见，阻碍了这些方法的进展。本文提出BIGbench，一个基于精心设计数据集的图像生成偏见统一基准。与现有基准不同，BIGbench将复杂偏见分类并评估为四个维度：偏见表现形式、偏见可见性、习得属性和受保护属性。此外，BIGbench应用先进的多模态大语言模型（MLLM），在保持高精度的同时实现全自动评估。我们应用BIGbench评估了八个近期通用T2I模型和三种去偏见方法，并进行了人工评估，其结果证明了BIGbench在图像对齐和识别各类偏见方面的有效性。此外，我们的研究还揭示了关于偏见的新研究方向，包括无关受保护属性的副作用和知识蒸馏的影响。我们的数据集和基准已向研究社区公开，以确保可复现性。