The unprecedented photorealistic results achieved by recent text-to-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases. In this work, we introduce three indicators to evaluate the realism, diversity and prompt-generation consistency of text-to-image generative systems when prompted to generate objects from across the world. Our indicators complement qualitative analysis of the broader impact of such systems by enabling automatic and efficient benchmarking of geographic disparities, an important step towards building responsible visual content creation systems. We use our proposed indicators to analyze potential geographic biases in state-of-the-art visual content creation systems and find that: (1) models have less realism and diversity of generations when prompting for Africa and West Asia than Europe, (2) prompting with geographic information comes at a cost to prompt-consistency and diversity of generated images, and (3) models exhibit more region-level disparities for some objects than others. Perhaps most interestingly, our indicators suggest that progress in image generation quality has come at the cost of real-world geographic representation. Our comprehensive evaluation constitutes a crucial step towards ensuring a positive experience of visual content creation for everyone.

翻译：近年来，文本到图像生成系统取得了前所未有的逼真效果，其作为即插即用的内容创作解决方案的应用日益广泛，因此理解其潜在偏见至关重要。本研究提出了三项指标，用于评估文本到图像生成系统在生成全球各地物体时的逼真度、多样性和提示-生成一致性。这些指标通过实现地理差异的自动化和高效基准测试，补充了对此类系统更广泛影响的定性分析，是构建负责任的视觉内容创作系统的重要一步。我们运用所提指标，分析了最先进视觉内容创作系统中的潜在地理偏见，发现：（1）在提示涉及非洲和西亚时，模型生成图像的逼真度和多样性低于欧洲；（2）在提示中加入地理信息会损害生成图像的提示一致性和多样性；（3）不同物体类别在区域层面的模型表现差异较大。或许最有趣的是，我们的指标表明，图像生成质量的进步是以牺牲现实世界地理表征为代价的。这项综合评估为确保每个人都能享有良好的视觉内容创作体验迈出了关键一步。