Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets. Code and checkpoints are available at https://github.com/microsoft/BridgeTower.

翻译：具有双塔架构的视觉-语言（Vision-Language，VL）模型近年来主导了视觉-语言表示学习。当前的VL模型要么使用轻量级单模态编码器，并在深度跨模态编码器中同时学习提取、对齐和融合两种模态，要么将来自深度预训练单模态编码器的最后一层单模态表示馈入顶层跨模态编码器。这两种方法都可能限制视觉-语言表示学习并影响模型性能。本文提出BridgeTower，它引入多个桥梁层，在单模态编码器的顶层与跨模态编码器的每一层之间建立连接。这使得在跨模态编码器中，能够对来自预训练单模态编码器的不同语义层次的视觉和文本表示进行有效的自底向上跨模态对齐与融合。仅使用400万张图像进行预训练，BridgeTower在多项下游视觉-语言任务上取得了最先进性能。特别是在VQAv2测试标准集上，BridgeTower在几乎不增加额外参数和计算成本的情况下，以相同的预训练数据取得了78.73%的准确率，较此前最先进模型METER提升1.09%。值得注意的是，进一步扩展模型规模后，BridgeTower达到了81.15%的准确率，超越了在数量级更大数据集上预训练的模型。代码和检查点已开源至https://github.com/microsoft/BridgeTower。