Recently, large-scale pre-trained vision-language models (e.g. CLIP and ALIGN) have demonstrated remarkable effectiveness in acquiring transferable visual representations. To leverage the valuable knowledge encoded within these models for downstream tasks, several fine-tuning approaches, including prompt tuning methods and adapter-based methods, have been developed to adapt vision-language models effectively with supervision. However, these methods rely on the availability of annotated samples, which can be labor-intensive and time-consuming to acquire, thus limiting scalability. To address this issue, in this work, we design an unsupervised fine-tuning approach for vision-language models called Unsupervised Prototype Adapter (UP-Adapter). Specifically, for the unannotated target datasets, we leverage the text-image aligning capability of CLIP to automatically select the most confident samples for each class. Utilizing these selected samples, we generate class prototypes, which serve as the initialization for the learnable prototype model. After fine-tuning, the prototype model prediction is combined with the original CLIP's prediction by a residual connection to perform downstream recognition tasks. Our extensive experimental results on image recognition and domain generalization show that the proposed unsupervised method outperforms 8-shot CoOp, 8-shot Tip-Adapter, and also the state-of-the-art UPL method by large margins.

翻译：近期，大规模预训练的视觉-语言模型（如CLIP和ALIGN）在获取可迁移视觉表征方面展现出显著效果。为将这些模型中的宝贵知识应用于下游任务，研究者开发了多种微调方法，包括提示调优方法和基于适配器的方法，从而在有监督条件下有效适配视觉-语言模型。然而，这些方法依赖于标注样本的可用性，而获取标注样本既耗时又耗费人力，限制了其可扩展性。为解决此问题，本文设计了一种面向视觉-语言模型的无监督微调方法，称为无监督原型适配器（UP-Adapter）。具体而言，针对未标注的目标数据集，我们利用CLIP的文本-图像对齐能力，自动为每个类别选择最置信的样本。基于这些选定样本生成类别原型，作为可学习原型模型的初始化。微调后，通过残差连接将原型模型预测与原始CLIP预测相结合，以执行下游识别任务。在图像识别和领域泛化上的大量实验结果表明，所提无监督方法在性能上大幅超越8-shot CoOp、8-shot Tip-Adapter以及当前最先进的UPL方法。