Benchmarks of the multilingual capabilities of text-to-image (T2I) models compare generated images prompted in a test language to an expected image distribution over a concept set. One such benchmark, "Conceptual Coverage Across Languages" (CoCo-CroLa), assesses the tangible noun inventory of T2I models by prompting them to generate pictures from a concept list translated to seven languages and comparing the output image populations. Unfortunately, we find that this benchmark contains translation errors of varying severity in Spanish, Japanese, and Chinese. We provide corrections for these errors and analyze how impactful they are on the utility and validity of CoCo-CroLa as a benchmark. We reassess multiple baseline T2I models with the revisions, compare the outputs elicited under the new translations to those conditioned on the old, and show that a correction's impactfulness on the image-domain benchmark results can be predicted in the text domain with similarity scores. Our findings will guide the future development of T2I multilinguality metrics by providing analytical tools for practical translation decisions.

翻译：多语言文本到图像（T2I）模型能力的基准测试比较的是：以测试语言生成的图像与概念集上的预期图像分布。其中一项基准测试“跨语言概念覆盖”（CoCo-CroLa）通过以下方式评估T2I模型的具体名词库存：用翻译成七种语言的概念列表提示模型生成图片，并比较输出的图像群体。不幸的是，我们发现该基准测试在西班牙语、日语和中文中存在严重程度不一的翻译错误。我们提供了这些错误的修正版本，并分析了它们对CoCo-CroLa作为基准测试的有效性和实用性的影响程度。我们使用修正后的翻译重新评估了多个基线T2I模型，比较了新翻译下产生的输出与基于旧翻译的输出，并证明文本域中相似度分数可以预测修正对图像域基准测试结果的影响程度。我们的发现将为未来T2I多语言度量指标的开发提供指导，并为实际翻译决策提供分析工具。