CLIP proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique entry in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multi-modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.

翻译：CLIP证明了对齐视觉与语言空间是无需显式训练即可解决许多视觉任务的关键，但需从零开始在大规模数据集上训练图像和文本编码器。LiT通过仅训练文本编码器并利用预训练视觉网络改进了这一点。本文证明，利用单领域编码器（在有无监督条件下训练）和少量图文对，完全无需任何训练即可构建公共空间。此外，我们的模型具有独特性质。最显著的是，更新训练样本后可在数秒内部署新版本。同时，公共空间中的表示具有高度可解释性——每个维度都对应输入与多模态数据集中唯一条目的相似度。标准零样本视觉基准实验展示了图文模型的典型迁移能力。总体而言，我们的方法为基础多模态模型提供了一个简单却出人意料的强大基线，对数据效率及检索在机器学习中的作用提出了重要问题。