Text-to-image (T2I) diffusion models achieve impressive photorealism by training on large-scale web data, but models inherit cultural biases and fail to depict underrepresented regions faithfully. Existing cultural benchmarks focus mainly on object-centric categories (e.g., food, attire, and architecture), overlooking the social and daily activities that more clearly reflect cultural norms. Few metrics exist for measuring cultural faithfulness. We introduce CULTIVate, a benchmark for evaluating T2I models on cross-cultural activities (e.g., greetings, dining, games, traditional dances, and cultural celebrations). CULTIVate spans 16 countries with 576 prompts and more than 19,000 images, and provides an explainable descriptor-based evaluation framework across multiple cultural dimensions, including background, attire, objects, and interactions. We propose four metrics to measure cultural alignment, hallucination, exaggerated elements, and diversity. Our findings reveal systematic disparities: models perform better for global north countries than for the global south, with distinct failure modes across T2I systems. Human studies confirm that our metrics correlate more strongly with human judgments than existing text-image metrics.
翻译:文本到图像(T2I)扩散模型通过在大规模网络数据上进行训练实现了令人印象深刻的照片级真实感,但这些模型继承了文化偏见,并且无法忠实地描绘代表性不足的地区。现有的文化基准主要关注以对象为中心的类别(例如食物、服饰和建筑),忽视了更能清晰反映文化规范的社会和日常活动。目前鲜有用于衡量文化忠实度的指标。我们提出了CULTIVate,这是一个用于评估T2I模型在跨文化活动(例如问候、用餐、游戏、传统舞蹈和文化庆典)上表现的基准。CULTIVate涵盖16个国家,包含576个提示词和超过19,000张图像,并提供了一个基于可解释描述符的评估框架,涵盖多个文化维度,包括背景、服饰、物体和互动。我们提出了四个指标来衡量文化对齐度、幻觉、夸张元素和多样性。我们的研究结果揭示了系统性的差异:模型对全球北方国家的表现优于对全球南方国家,并且不同的T2I系统表现出不同的失败模式。人类研究证实,与现有的文本-图像指标相比,我们的指标与人类判断的相关性更强。