Text-to-Image (T2I) models are being increasingly adopted in diverse global communities where they create visual representations of their unique cultures. Current T2I benchmarks primarily focus on faithfulness, aesthetics, and realism of generated images, overlooking the critical dimension of cultural competence. In this work, we introduce a framework to evaluate cultural competence of T2I models along two crucial dimensions: cultural awareness and cultural diversity, and present a scalable approach using a combination of structured knowledge bases and large language models to build a large dataset of cultural artifacts to enable this evaluation. In particular, we apply this approach to build CUBE (CUltural BEnchmark for Text-to-Image models), a first-of-its-kind benchmark to evaluate cultural competence of T2I models. CUBE covers cultural artifacts associated with 8 countries across different geo-cultural regions and along 3 concepts: cuisine, landmarks, and art. CUBE consists of 1) CUBE-1K, a set of high-quality prompts that enable the evaluation of cultural awareness, and 2) CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding to evaluate cultural diversity. We also introduce cultural diversity as a novel T2I evaluation component, leveraging quality-weighted Vendi score. Our evaluations reveal significant gaps in the cultural awareness of existing models across countries and provide valuable insights into the cultural diversity of T2I outputs for under-specified prompts. Our methodology is extendable to other cultural regions and concepts, and can facilitate the development of T2I models that better cater to the global population.

翻译：文本到图像（T2I）模型正日益被全球不同社群所采用，用于生成其独特文化的视觉表征。当前的T2I基准测试主要关注生成图像的忠实度、美学质量和真实感，而忽视了文化能力这一关键维度。在本研究中，我们引入了一个框架，从文化意识与文化多样性这两个关键维度评估T2I模型的文化能力，并提出了一种可扩展的方法，结合结构化知识库与大型语言模型构建大规模文化器物数据集以支持此项评估。具体而言，我们应用该方法构建了CUBE（文本到图像模型文化基准），这是首个用于评估T2I模型文化能力的基准。CUBE涵盖来自不同地理文化区域的8个国家，并围绕烹饪、地标和艺术这3个概念相关的文化器物。CUBE包含：1）CUBE-1K，一组用于评估文化意识的高质量提示词；2）CUBE-CSpace，一个作为评估文化多样性基础的大型文化器物数据集。我们还引入文化多样性作为新的T2I评估指标，采用质量加权的Vendi分数进行计算。我们的评估揭示了现有模型在不同国家文化意识方面存在的显著差距，并为理解T2I模型在非限定提示下输出结果的文化多样性提供了重要见解。该方法可扩展至其他文化区域与概念，有助于开发更能满足全球用户需求的T2I模型。