Cross-modal alignment is essential for vision-language pre-training (VLP) models to learn the correct corresponding information across different modalities. For this purpose, inspired by the success of masked language modeling (MLM) tasks in the NLP pre-training area, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-to-local alignment. However, most of them pay little attention to the global semantic features generated for the masked data, resulting in a limited cross-modal alignment ability of global representations. Therefore, in this paper, we propose a novel Semantic Completion Learning (SCL) task, complementary to existing masked modeling tasks, to facilitate global-to-local alignment. Specifically, the SCL task complements the missing semantics of masked data by capturing the corresponding information from the other modality, promoting learning more representative global features which have a great impact on the performance of downstream tasks. Moreover, we present a flexible vision encoder, which enables our model to perform image-text and video-text multimodal tasks simultaneously. Experimental results show that our proposed method obtains state-of-the-art performance on various vision-language benchmarks, such as visual question answering, image-text retrieval, and video-text retrieval.

翻译：跨模态对齐对于视觉语言预训练（VLP）模型学习不同模态间的正确对应信息至关重要。为此，受自然语言处理预训练领域遮蔽语言建模（MLM）任务成功的启发，VLP领域也提出了多种遮蔽建模任务以进一步促进跨模态交互。先前遮蔽建模任务的核心思想是基于可见上下文重构被遮蔽的标记，从而学习局部到局部的对齐。然而，大多数任务很少关注为被遮蔽数据生成的全局语义特征，导致全局表征的跨模态对齐能力有限。因此，本文提出了一种新颖的语义补全学习（SCL）任务，作为现有遮蔽建模任务的补充，以促进全局到局部的对齐。具体而言，SCL任务通过从另一模态捕获对应信息来补全被遮蔽数据的缺失语义，从而促进更具代表性的全局特征的学习，这些特征对下游任务的性能有重要影响。此外，我们提出了一种灵活的视觉编码器，使模型能够同时执行图像-文本和视频-文本多模态任务。实验结果表明，我们提出的方法在多种视觉语言基准任务（如视觉问答、图像-文本检索和视频-文本检索）上取得了最先进的性能。