Numerous visio-linguistic (V+L) representation learning methods have been developed, yet existing datasets do not adequately evaluate the extent to which they represent visual and linguistic concepts in a unified space. We propose several novel evaluation settings for V+L models, including cross-modal transfer. Furthermore, existing V+L benchmarks often report global accuracy scores on the entire dataset, making it difficult to pinpoint the specific reasoning tasks that models fail and succeed at. We present TraVLR, a synthetic dataset comprising four V+L reasoning tasks. TraVLR's synthetic nature allows us to constrain its training and testing distributions along task-relevant dimensions, enabling the evaluation of out-of-distribution generalisation. Each example in TraVLR redundantly encodes the scene in two modalities, allowing either to be dropped or added during training or testing without losing relevant information. We compare the performance of four state-of-the-art V+L models, finding that while they perform well on test examples from the same modality, they all fail at cross-modal transfer and have limited success accommodating the addition or deletion of one modality. We release TraVLR as an open challenge for the research community.

翻译：大量視覺-語言（V+L）表徵學習方法已被提出，然而現有數據集未能充分評估其將視覺與語言概念統一表徵於共同空間的能力。我們為V+L模型提出了多項新穎的評估設定，包括跨模態遷移。此外，現有V+L基準通常報告整個數據集的全局準確率分數，難以精確定位模型成敗的具體推理任務。本文提出TraVLR——一個由四項V+L推理任務組成的合成數據集。TraVLR的合成特性允許我們沿任務相關維度約束其訓練與測試分佈，從而實現對分佈外泛化能力的評估。TraVLR中的每個樣本均以兩種模態冗餘編碼場景信息，因此在訓練或測試過程中可任意移除或添加任一模態而不損失相關信息。我們比較了四種先進V+L模型的性能，發現雖然它們在同模態測試樣本上表現良好，但均無法完成跨模態遷移，且對單一模態的增刪處理能力有限。我們將TraVLR作為開放式挑戰發布給研究社區。