Language models have been supervised with both language-only objective and visual grounding in existing studies of visual-grounded language learning. However, due to differences in the distribution and scale of visual-grounded datasets and language corpora, the language model tends to mix up the context of the tokens that occurred in the grounded data with those that do not. As a result, during representation learning, there is a mismatch between the visual information and the contextual meaning of the sentence. To overcome this limitation, we propose GroundedBERT - a grounded language learning method that enhances the BERT representation with visually grounded information. GroundedBERT comprises two components: (i) the original BERT which captures the contextual representation of words learned from the language corpora, and (ii) a visual grounding module which captures visual information learned from visual-grounded datasets. Moreover, we employ Optimal Transport (OT), specifically its partial variant, to solve the fractional alignment problem between the two modalities. Our proposed method significantly outperforms the baseline language models on various language tasks of the GLUE and SQuAD datasets.

翻译：在现有视觉语言学习研究中，语言模型同时受到纯语言目标和视觉基础这两个监督信号的约束。然而，由于视觉基础数据集与语言语料库在分布和规模上存在差异，语言模型容易混淆在基础数据中出现过的词与未出现词的上下文。这种差异导致表示学习过程中视觉信息与句子的语境含义之间出现不匹配。为克服这一局限，我们提出GroundedBERT——一种利用视觉基础信息增强BERT表示的基于语言学习方法。GroundedBERT包含两个组件：（i）原始BERT，用于捕捉从语言语料库中学习到的单词上下文表示；（ii）视觉基础模块，用于捕捉从视觉基础数据集中学习到的视觉信息。此外，我们采用最优传输（Optimal Transport，OT）的部分变体来解决两种模态间的部分对齐问题。在GLUE和SQuAD数据集的各种语言任务中，我们提出的方法显著优于基线语言模型。