Vision-language pre-training (VLP) methods are blossoming recently, and its crucial goal is to jointly learn visual and textual features via a transformer-based architecture, demonstrating promising improvements on a variety of vision-language tasks. Prior arts usually focus on how to align visual and textual features, but strategies for improving the robustness of model and speeding up model convergence are left insufficiently explored. In this paper, we propose a novel method ViLTA, comprising of two components to further facilitate the model to learn fine-grained representations among image-text pairs. For Masked Language Modeling (MLM), we propose a cross-distillation method to generate soft labels to enhance the robustness of model, which alleviates the problem of treating synonyms of masked words as negative samples in one-hot labels. For Image-Text Matching (ITM), we leverage the current language encoder to synthesize hard negatives based on the context of language input, encouraging the model to learn high-quality representations by increasing the difficulty of the ITM task. By leveraging the above techniques, our ViLTA can achieve better performance on various vision-language tasks. Extensive experiments on benchmark datasets demonstrate that the effectiveness of ViLTA and its promising potential for vision-language pre-training.

翻译：视觉-语言预训练（VLP）方法近期蓬勃发展，其核心目标是通过基于Transformer的架构联合学习视觉和文本特征，并在多种视觉-语言任务上展现出显著改进。现有研究通常聚焦于如何对齐视觉与文本特征，而提升模型鲁棒性及加速模型收敛的策略仍未被充分探索。本文提出一种名为ViLTA的新方法，包含两个组件以进一步促进模型学习图像-文本对间的细粒度表示。对于掩码语言建模（MLM），我们提出一种交叉蒸馏方法生成软标签以增强模型鲁棒性，从而缓解独热（one-hot）标签中将掩码词的同义词视为负样本的问题。对于图像-文本匹配（ITM），我们利用当前语言编码器基于语言输入上下文合成难负样本，通过增加ITM任务难度激励模型学习高质量表示。凭借上述技术，我们的ViLTA能够在多种视觉-语言任务上取得更优性能。在基准数据集上的大量实验证明了ViLTA的有效性及其在视觉-语言预训练中的巨大潜力。