Human language is grounded on multimodal knowledge including visual knowledge like colors, sizes, and shapes. However, current large-scale pre-trained language models rely on text-only self-supervised training with massive text data, which precludes them from utilizing relevant visual information when necessary. To address this, we propose a novel pre-training framework, named VaLM, to Visually-augment text tokens with retrieved relevant images for Language Modeling. Specifically, VaLM builds on a novel latent text-image alignment method via an image retrieval module to fetch corresponding images given a textual context. With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending to both text context and visual knowledge in images. We evaluate VaLM on various visual knowledge-intensive commonsense reasoning tasks, which require visual information to excel. The experimental results illustrate that VaLM outperforms all strong language-only and vision-language baselines with substantial gains in reasoning object commonsense including color, size, and shape. Our code is available at https://github.com/Victorwz/VaLM.

翻译：人类语言基于多模态知识，包括颜色、大小和形状等视觉知识。然而，当前大规模预训练语言模型仅依赖海量文本数据的纯文本自监督训练，这使其在必要时无法利用相关视觉信息。为解决此问题，我们提出一种新型预训练框架VaLM（视觉增强语言模型），通过检索相关图像来视觉增强文本标记。具体而言，VaLM借助图像检索模块，基于一种新颖的隐式文本-图像对齐方法，根据文本上下文获取对应图像。利用视觉增强的上下文后，VaLM通过视觉知识融合层，同时关注文本上下文和图像中的视觉知识，实现多模态基础语言建模。我们在需要视觉信息才能获得出色表现的各类视觉知识密集型常识推理任务上评估了VaLM。实验结果表明，VaLM在所有仅文本和视觉-语言强基线上均取得显著优势，尤其在推理物体常识（包括颜色、大小和形状）方面表现突出。我们的代码已开源在https://github.com/Victorwz/VaLM。