We introduce Three Towers (3T), a flexible method to improve the contrastive learning of vision-language models by incorporating pretrained image classifiers. While contrastive models are usually trained from scratch, LiT (Zhai et al., 2022) has recently shown performance gains from using pretrained classifier embeddings. However, LiT directly replaces the image tower with the frozen embeddings, excluding any potential benefits of contrastively training the image tower. With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training. To achieve this, we introduce a third tower that contains the frozen pretrained embeddings, and we encourage alignment between this third tower and the main image-text towers. Empirically, 3T consistently improves over LiT and the CLIP-style from-scratch baseline for retrieval tasks. For classification, 3T reliably improves over the from-scratch baseline, and while it underperforms relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k and Places365 pretraining.

翻译：我们提出三塔（Three Towers，3T）方法，通过整合预训练图像分类器，实现视觉-语言模型对比学习的灵活改进。通常对比学习模型是从头开始训练的，而LiT（Zhai等，2022）近期展示了使用预训练分类器嵌入能够带来性能提升。但LiT直接用冻结嵌入替换图像塔，排除了对图像塔进行对比训练的任何潜在优势。通过3T，我们提出一种更灵活的策略，使图像塔既能受益于预训练嵌入，又能受益于对比训练。为此，我们引入第三塔（包含冻结的预训练嵌入），并促进第三塔与主图像-文本塔之间的对齐。实验表明，3T在检索任务中持续优于LiT和CLIP风格从头训练的基线方法。在分类任务中，3T可靠地超越从头训练基线；虽然对于JFT预训练模型其性能低于LiT，但在ImageNet-21k和Places365预训练场景下，3T优于LiT。