Large-scale vision-language models (VLMs) such as CLIP exhibit strong zero-shot generalization, but adapting them to downstream tasks typically requires costly labeled data. Existing unsupervised self-training methods rely on pseudo-labeling, yet often suffer from unreliable confidence filtering, confirmation bias, and underutilization of low-confidence samples. We propose Collaborative Fine-Tuning (CoFT), an unsupervised adaptation framework that leverages unlabeled data through a dual-model, cross-modal collaboration mechanism. CoFT introduces a dual-prompt learning strategy with positive and negative textual prompts to explicitly model pseudo-label cleanliness in a sample-dependent manner, removing the need for hand-crafted thresholds or noise assumptions. The negative prompt also regularizes lightweight visual adaptation modules, improving robustness under noisy supervision. CoFT employs a two-phase training scheme, transitioning from parameter-efficient fine-tuning on high-confidence samples to full fine-tuning guided by collaboratively filtered pseudo-labels. Building on CoFT, CoFT+ further enhances adaptation via iterative fine-tuning, momentum contrastive learning, and LLM-generated prompts. Extensive experiments demonstrate consistent gains over existing unsupervised methods and even few-shot supervised baselines.

翻译：大规模视觉语言模型（如CLIP）展现出强大的零样本泛化能力，但将其适配至下游任务通常需要昂贵的标注数据。现有的无监督自训练方法依赖伪标注技术，但常受限于不可靠的置信度筛选、确认偏误及低置信度样本利用不足等问题。本文提出协同微调框架，该无监督适配框架通过双模型跨模态协作机制利用未标注数据。CoFT引入包含正向与负向文本提示的双提示学习策略，以样本依赖的方式显式建模伪标注洁净度，无需人工设定阈值或噪声假设。负向提示同时可正则化轻量级视觉适配模块，提升噪声监督下的鲁棒性。CoFT采用两阶段训练方案：第一阶段对高置信度样本进行参数高效微调，第二阶段在协同筛选的伪标注指导下进行全参数微调。基于CoFT框架，CoFT+通过迭代微调、动量对比学习与大语言模型生成提示进一步强化适配能力。大量实验表明，该方法在多个基准测试中持续优于现有无监督方法，甚至超越少样本监督基线。