Bias in text-to-image (T2I) models can propagate unfair social representations and may be used to aggressively market ideas or push controversial agendas. Existing T2I model bias evaluation methods only focus on social biases. We look beyond that and instead propose an evaluation methodology to quantify general biases in T2I generative models, without any preconceived notions. We assess four state-of-the-art T2I models and compare their baseline bias characteristics to their respective variants (two for each), where certain biases have been intentionally induced. We propose three evaluation metrics to assess model biases including: (i) Distribution bias, (ii) Jaccard hallucination and (iii) Generative miss-rate. We conduct two evaluation studies, modelling biases under general, and task-oriented conditions, using a marketing scenario as the domain for the latter. We also quantify social biases to compare our findings to related works. Finally, our methodology is transferred to evaluate captioned-image datasets and measure their bias. Our approach is objective, domain-agnostic and consistently measures different forms of T2I model biases. We have developed a web application and practical implementation of what has been proposed in this work, which is at https://huggingface.co/spaces/JVice/try-before-you-bias. A video series with demonstrations is available at https://www.youtube.com/channel/UCk-0xyUyT0MSd_hkp4jQt1Q

翻译：文本到图像（T2I）模型中的偏见可能传播不公平的社会表征，并被用于激进营销观点或推动有争议的议程。现有T2I模型偏见评估方法仅关注社会偏见。我们突破这一局限，提出一种评估方法论，旨在无任何先入为主观念的情况下量化T2I生成模型中的通用偏见。我们评估了四种最先进的T2I模型，并将其基线偏见特征与各自变体（每种模型对应两个变体，其中已刻意引入特定偏见）进行对比。我们提出三项评估指标用以衡量模型偏见，包括：（i）分布偏见、（ii）Jaccard幻觉度以及（iii）生成漏检率。我们开展两项评估研究，分别在通用条件和面向任务条件下建模偏见，并以营销场景作为后者的研究领域。同时，我们量化社会偏见，以便将研究结果与已有文献进行对比。最后，我们将所提方法论迁移至带标题图像数据集的评估中，以衡量其偏见。我们的方法具有客观性、领域无关性，并能一致地测量不同形式的T2I模型偏见。基于本工作成果，我们开发了一个网络应用程序及实用化实现（访问地址：https://huggingface.co/spaces/JVice/try-before-you-bias），并提供了包含演示的视频系列（链接：https://www.youtube.com/channel/UCk-0xyUyT0MSd_hkp4jQt1Q）。