Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. S-CLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs.

翻译：视觉-语言模型（如对比语言-图像预训练模型CLIP）在自然图像领域取得了显著成果。然而，当应用于遥感等专业领域时，这些模型往往表现不佳，且由于可用的图像-文本对数量有限，适应此类领域充满挑战。为此，我们提出S-CLIP——一种利用额外未配对图像来训练CLIP的半监督学习方法。S-CLIP采用两种专为对比学习和语言模态设计的伪标签策略：其一是通过求解未配对图像与配对图像之间的最优传输问题，获得配对图像标注的合成结果作为字幕级伪标签；其二是利用最近邻配对图像标注中的关键词作为关键词级伪标签，通过假设监督信号为候选标签集（而非精确标签）的部分标签学习进行训练。结合这些目标后，S-CLIP在仅有少量图像-文本对的情况下显著提升了CLIP的训练效果——这一优势在遥感、时尚、科学图像及漫画等多个专业领域得到验证。例如，在遥感基准测试中，S-CLIP使CLIP的零样本分类性能提升10%、图像-文本检索性能提升4%，在仅使用三倍更少图像-文本对的条件下即可达到监督式CLIP的性能水平。