We propose a simple pairwise sigmoid loss for image-text pre-training. Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. With only four TPUv4 chips, we can train a Base CLIP model at 4k batch size and a Large LiT model at 20k batch size, the latter achieves 84.5% ImageNet zero-shot accuracy in two days. This disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

翻译：我们提出了一种用于图像-文本预训练的简单成对sigmoid损失函数。与采用softmax归一化的标准对比学习不同，sigmoid损失函数仅作用于图像-文本对，无需通过全局视角对成对相似度进行归一化。该损失函数既能支持批处理规模的进一步扩大，又能在较小批处理规模下表现更优。仅需四块TPUv4芯片，我们就能以4k批处理规模训练Base CLIP模型，以20k批处理规模训练Large LiT模型，后者在两天内实现了84.5%的ImageNet零样本准确率。这种批处理规模与损失函数的解耦，使我们能够进一步研究样本数量与配对数量以及负样本-正样本比例的影响。最后，我们将批处理规模推至极值（百万级别），发现增加批处理规模的收益会迅速衰减，而32k的合理批处理规模已足够。我们希望这项研究能激励人们进一步探索提升语言-图像预训练质量与效率的方法。