We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We release our models at https://github.com/google-research/big_vision and hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

翻译：我们提出了一种用于语言-图像预训练的简单成对Sigmoid损失（SigLIP）。与采用Softmax归一化的标准对比学习不同，Sigmoid损失仅基于图像-文本对进行操作，无需全局视角的成对相似度归一化。该损失函数既能支持更大规模的批处理量扩展，又能在小批量情况下表现更优。结合锁定图像微调技术，我们仅使用四个TPUv4芯片，在两天内训练出SigLiT模型，其ImageNet零样本准确率达到84.5%。损失函数与批处理量的解耦使我们能够研究样本与配对、负样本与正样本比例的影响。最后，我们将批处理量推至百万量级的极限，发现增大批处理量的收益迅速衰减，32k的批处理量已足够有效。我们在https://github.com/google-research/big_vision发布模型，期望本研究能推动对语言-图像预训练质量与效率的提升探索。