We introduce Three Towers (3T), a flexible method to improve the contrastive learning of vision-language models by incorporating pretrained image classifiers. While contrastive models are usually trained from scratch, LiT (Zhai et al., 2022) has recently shown performance gains from using pretrained classifier embeddings. However, LiT directly replaces the image tower with the frozen embeddings, excluding any potential benefits of contrastively training the image tower. With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training. To achieve this, we introduce a third tower that contains the frozen pretrained embeddings, and we encourage alignment between this third tower and the main image-text towers. Empirically, 3T consistently improves over LiT and the CLIP-style from-scratch baseline for retrieval tasks. For classification, 3T reliably improves over the from-scratch baseline, and while it underperforms relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k and Places365 pretraining.

翻译：我们提出三塔结构（3T），一种通过引入预训练图像分类器来改进视觉-语言模型对比学习的灵活方法。虽然对比模型通常从零开始训练，但LiT（Zhai等，2022）最近展示了使用预训练分类器嵌入带来的性能提升。然而，LiT直接用冻结嵌入替换图像塔，排除了对图像塔进行对比训练可能带来的任何益处。通过3T，我们提出一种更灵活的策略，使图像塔既能受益于预训练嵌入，又能受益于对比训练。为此，我们引入包含冻结预训练嵌入的第三塔，并鼓励该第三塔与主图像-文本塔之间的对齐。实验表明，3T在检索任务上持续优于LiT和基于CLIP风格的从头训练基线。对于分类任务，3T可靠地优于从头训练基线；虽然在使用JFT预训练模型时其性能逊于LiT，但在ImageNet-21k和Places365预训练场景中优于LiT。