Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones, especially in the context of large multimodal models. At the same time, image captioning on this type of data is commonly considered an inferior pretraining strategy. In this paper, we perform a fair comparison of these two pretraining strategies, carefully matching training data, compute, and model capacity. Using a standard encoder-decoder transformer, we find that captioning alone is surprisingly effective: on classification tasks, captioning produces vision encoders competitive with contrastively pretrained encoders, while surpassing them on vision & language tasks. We further analyze the effect of the model architecture and scale, as well as the pretraining data on the representation quality, and find that captioning exhibits the same or better scaling behavior along these axes. Overall our results show that plain image captioning is a more powerful pretraining strategy than was previously believed.

翻译：从网络获取的图像-文本对进行对比预训练是最流行的大规模视觉骨干网络预训练策略之一，尤其是在大型多模态模型的背景下。与此同时，基于此类数据的图像描述通常被视为较差的预训练策略。在本文中，我们对这两种预训练策略进行了公平比较，谨慎匹配了训练数据、计算资源和模型容量。使用标准编码器-解码器Transformer架构，我们发现仅凭图像描述就出人意料地有效：在分类任务上，图像描述生成的视觉编码器与对比预训练的编码器性能相当，同时在视觉与语言任务上超越了后者。我们进一步分析了模型架构与规模以及预训练数据对表征质量的影响，并发现图像描述在这些维度上表现出相同或更优的扩展行为。总体而言，我们的结果表明，单纯图像描述是一种比先前认知更强大的预训练策略。