Computer vision often treats perception as objective, and this assumption gets reflected in the way that datasets are collected and models are trained. For instance, image descriptions in different languages are typically assumed to be translations of the same semantic content. However, work in cross-cultural psychology and linguistics has shown that individuals differ in their visual perception depending on their cultural background and the language they speak. In this paper, we demonstrate significant differences in semantic content across languages in both dataset and model-produced captions. When data is multilingual as opposed to monolingual, captions have higher semantic coverage on average, as measured by scene graph, embedding, and linguistic complexity. For example, multilingual captions have on average 21.8% more objects, 24.5% more relations, and 27.1% more attributes than a set of monolingual captions. Moreover, models trained on content from different languages perform best against test data from those languages, while those trained on multilingual content perform consistently well across all evaluation data compositions. Our research provides implications for how diverse modes of perception can improve image understanding.

翻译：计算机视觉常将感知视为客观过程，这一假设体现在数据集构建与模型训练方式中。例如，不同语言的图像描述通常被假定为同一语义内容的翻译。然而，跨文化心理学与语言学研究表明，个体的视觉感知会因其文化背景及所用语言而存在显著差异。本文证明，在数据集与模型生成的描述中，不同语言的语义内容存在显著差异。相较于单语言数据，多语言数据的描述在场景图、嵌入表征及语言复杂度等指标上展现出更高的平均语义覆盖率。例如，多语言描述相较于单语言描述集，平均包含21.8%更多的物体、24.5%更多的关系及27.1%更多的属性。此外，基于特定语言内容训练的模型在该语言测试数据上表现最佳，而基于多语言内容训练的模型则在所有评估数据构成中保持稳定优异的性能。本研究揭示了多样化感知模式对提升图像理解的启示。