Vision-Language Pre-training (VLP) methods based on object detection enjoy the rich knowledge of fine-grained object-text alignment but at the cost of computationally expensive inference. Recent Visual-Transformer (ViT)-based approaches circumvent this issue while struggling with long visual sequences without detailed cross-modal alignment information. This paper introduces a ViT-based VLP technique that efficiently incorporates object information through a novel patch-text alignment mechanism. Specifically, we convert object-level signals into patch-level ones and devise a Patch-Text Alignment pre-training task (PTA) to learn a text-aware patch detector. By using off-the-shelf delicate object annotations in 5\% training images, we jointly train PTA with other conventional VLP objectives in an end-to-end manner, bypassing the high computational cost of object detection and yielding an effective patch detector that accurately detects text-relevant patches, thus considerably reducing patch sequences and accelerating computation within the ViT backbone. Our experiments on a variety of widely-used benchmarks reveal that our method achieves a speedup of nearly 88\% compared to prior VLP models while maintaining competitive or superior performance on downstream tasks with similar model size and data scale.

翻译：基于目标检测的视觉-语言预训练（VLP）方法虽能充分利用细粒度对象-文本对齐的丰富知识，但推理时需承担高昂计算成本。近期基于视觉Transformer（ViT）的方法规避了该问题，却在处理长视觉序列时缺乏细致的跨模态对齐信息。本文提出一种基于ViT的VLP技术，通过新颖的补丁-文本对齐机制高效融入对象信息。具体而言，我们将对象级信号转换为补丁级信号，并设计补丁-文本对齐预训练任务（PTA）以学习文本感知的补丁检测器。通过仅使用5%训练图像中的现成精细对象标注，我们以端到端方式联合训练PTA与其他传统VLP目标，规避了目标检测的高计算开销，从而生成能准确检测文本相关补丁的高效检测器，大幅缩短补丁序列并加速ViT骨干网络的计算。在各类广泛使用的基准测试中，我们的方法在保持与先前VLP模型相似的模型规模和数据规模时，实现了近88%的推理加速，同时在下游任务中保持相当或更优的性能。