One challenge in text-to-image (T2I) generation is the inadvertent reflection of culture gaps present in the training data, which signifies the disparity in generated image quality when the cultural elements of the input text are rarely collected in the training set. Although various T2I models have shown impressive but arbitrary examples, there is no benchmark to systematically evaluate a T2I model's ability to generate cross-cultural images. To bridge the gap, we propose a Challenging Cross-Cultural (C3) benchmark with comprehensive evaluation criteria, which can assess how well-suited a model is to a target culture. By analyzing the flawed images generated by the Stable Diffusion model on the C3 benchmark, we find that the model often fails to generate certain cultural objects. Accordingly, we propose a novel multi-modal metric that considers object-text alignment to filter the fine-tuning data in the target culture, which is used to fine-tune a T2I model to improve cross-cultural generation. Experimental results show that our multi-modal metric provides stronger data selection performance on the C3 benchmark than existing metrics, in which the object-text alignment is crucial. We release the benchmark, data, code, and generated images to facilitate future research on culturally diverse T2I generation (https://github.com/longyuewangdcu/C3-Bench).

翻译：文本到图像生成（T2I）面临的一个挑战是训练数据中无意反映出的文化差距，即当输入文本的文化元素在训练集中鲜有出现时，生成的图像质量会出现差异。尽管各类T2I模型已展现出令人印象深刻但任意的生成案例，目前仍缺乏系统性评估模型跨文化图像生成能力的基准。为弥补这一空白，我们提出具有综合评价标准的挑战性跨文化（C3）基准，可评估模型对目标文化的适配程度。通过分析Stable Diffusion模型在C3基准上生成的缺陷图像，我们发现该模型常无法生成特定文化物品。据此，我们提出一种考虑对象-文本对齐的新型多模态指标，用于筛选目标文化中的微调数据，进而对T2I模型进行微调以提升跨文化生成能力。实验结果表明，在C3基准上，我们的多模态指标相比现有指标展现出更强的数据筛选性能，其中对象-文本对齐至关重要。为促进文化多样性T2I生成研究，我们公开了基准、数据、代码及生成图像（https://github.com/longyuewangdcu/C3-Bench）。