The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

翻译：视觉与语言预训练因大规模模型的端到端训练而成本日益高昂。本文提出BLIP-2，一种通用高效的预训练策略，通过现成的冻结预训练图像编码器和冻结大语言模型来引导视觉-语言预训练。BLIP-2采用轻量级Querying Transformer弥合模态差异，该模型分两阶段进行预训练：第一阶段从冻结图像编码器引导视觉-语言表示学习，第二阶段从冻结语言模型引导视觉到语言的生成式学习。尽管BLIP-2的可训练参数远少于现有方法，其在各类视觉-语言任务上仍取得了最先进性能。例如，我们的模型在零样本VQAv2任务上以54倍更少的可训练参数超越Flamingo80B达8.7%。我们同时展示了模型遵循自然语言指令进行零样本图像到文本生成的新兴能力。