Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets. Code and checkpoints are available at https://github.com/microsoft/BridgeTower.

翻译：近年来，采用双塔架构的视觉-语言（Vision-Language，VL）模型在视觉-语言表征学习中占据主导地位。当前的VL模型要么使用轻量级单模态编码器，并在深度跨模态编码器中同时学习提取、对齐和融合两种模态，要么将深度预训练单模态编码器的最后一层表示输入到顶层跨模态编码器中。这两种方法均可能限制视觉-语言表征学习并影响模型性能。本文提出BridgeTower模型，该模型引入多个桥接层，在单模态编码器的顶层与跨模态编码器的每一层之间建立连接。这使跨模态编码器能够有效实现预训练单模态编码器不同语义层次的视觉与文本表示之间的自底向上跨模态对齐与融合。仅使用400万张图像进行预训练，BridgeTower即在多种下游视觉-语言任务中取得了最先进的性能。具体而言，在VQAv2测试标准集上，BridgeTower达到78.73%的准确率，在相同预训练数据及几乎可忽略的额外参数与计算成本下，比此前最先进模型METER高出1.09%。值得注意的是，进一步扩展模型规模后，BridgeTower的准确率达到81.15%，超越了使用数量级更大数据集预训练的模型。代码及模型检查点已开源至https://github.com/microsoft/BridgeTower。