During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.

翻译：在过去两年间，视觉-语言预训练在多个下游任务中取得了显著成功。然而，获取完全互斥的高质量图像-文本对仍是一项艰巨挑战，且常用数据集中存在噪声。针对这一问题，我们提出SoftCLIP——一种通过引入基于细粒度模态内自相似性生成的软化目标来放宽严格的一对一约束、实现柔性跨模态对齐的新方法。模态内引导具有指示性，能够使两个模态对具有局部相似性，并建模两者间的多对多关系。此外，由于正样本在软化目标分布中仍占主导地位，我们在该分布中分离负样本，以进一步强化跨模态学习中的负样本关系对齐。大量实验证明了SoftCLIP的有效性。特别地，在ImageNet零样本分类任务中，以CC3M/CC12M作为预训练数据集时，SoftCLIP相较于CLIP基线分别在top-1准确率上提升6.8%/7.2%。