Recently, large-scale pre-trained vision-language models (e.g. CLIP and ALIGN) have demonstrated remarkable effectiveness in acquiring transferable visual representations. To leverage the valuable knowledge encoded within these models for downstream tasks, several fine-tuning approaches, including prompt tuning methods and adapter-based methods, have been developed to adapt vision-language models effectively with supervision. However, these methods rely on the availability of annotated samples, which can be labor-intensive and time-consuming to acquire, thus limiting scalability. To address this issue, in this work, we design an unsupervised fine-tuning approach for vision-language models called Unsupervised Prototype Adapter (UP-Adapter). Specifically, for the unannotated target datasets, we leverage the text-image aligning capability of CLIP to automatically select the most confident samples for each class. Utilizing these selected samples, we generate class prototypes, which serve as the initialization for the learnable prototype model. After fine-tuning, the prototype model prediction is combined with the original CLIP's prediction by a residual connection to perform downstream recognition tasks. Our extensive experimental results on image recognition and domain generalization show that the proposed unsupervised method outperforms 8-shot CoOp, 8-shot Tip-Adapter, and also the state-of-the-art UPL method by large margins.

翻译：近期，大规模预训练的视觉-语言模型（如CLIP和ALIGN）在获取可迁移视觉表征方面展现出了显著的有效性。为利用这些模型中编码的宝贵知识处理下游任务，研究者设计了多种微调方法，包括提示调整方法和基于适配器的方法，从而在有监督条件下有效适配视觉-语言模型。然而，这些方法依赖已标注样本的可用性，而获取标注样本既费时又费力，限制了其可扩展性。为解决这一问题，本文设计了一种无监督微调方法——无监督原型适配器（Unsupervised Prototype Adapter, UP-Adapter）。具体而言，对于未标注的目标数据集，我们利用CLIP的文本-图像对齐能力自动为每个类别挑选最可信的样本。借助这些选中的样本，我们生成类别原型，并将其作为可学习原型模型的初始化参数。微调后，原型模型的预测结果通过残差连接与原始CLIP的预测结果相结合，用于执行下游识别任务。我们在图像识别和领域泛化任务上的大量实验结果表明，所提出的无监督方法大幅优于8-shot CoOp、8-shot Tip-Adapter以及当前最先进的UPL方法。