Compositionality is a common property in many modalities including natural languages and images, but the compositional generalization of multi-modal models is not well-understood. In this paper, we identify two sources of visual-linguistic compositionality: linguistic priors and the interplay between images and texts. We show that current attempts to improve compositional generalization rely on linguistic priors rather than on information in the image. We also propose a new metric for compositionality without such linguistic priors.

翻译：组合性是包括自然语言和图像在内的多种模态的常见属性，但多模态模型的组合泛化能力尚未得到充分理解。本文识别出视觉-语言组合性的两个来源：语言先验以及图像与文本之间的相互作用。我们表明，当前改进组合泛化的尝试依赖于语言先验而非图像中的信息。同时，我们提出了一种无需此类语言先验的组合性新指标。