We introduce ProLIP, a simple and architecture-agnostic method for adapting contrastively pretrained vision-language models, such as CLIP, to few-shot classification. ProLIP fine-tunes the vision encoder's projection matrix with Frobenius norm regularization on its deviation from the pretrained weights. It achieves state-of-the-art performance on 11 few-shot classification benchmarks under both ``few-shot validation'' and ``validation-free'' settings. Moreover, by rethinking the non-linear CLIP-Adapter through ProLIP's lens, we design a Regularized Linear Adapter (RLA) that performs better, requires no hyperparameter tuning, is less sensitive to learning rate values, and offers an alternative to ProLIP in black-box scenarios where model weights are inaccessible. Beyond few-shot classification, ProLIP excels in cross-dataset transfer, domain generalization, base-to-new class generalization, and test-time adaptation--where it outperforms prompt tuning while being an order of magnitude faster to train. Code is available at https://github.com/astra-vision/ProLIP .

翻译：我们提出了ProLIP，一种简单且架构无关的方法，用于将对比预训练的视觉-语言模型（如CLIP）适配到少样本分类任务中。ProLIP通过Frobenius范数正则化视觉编码器投影矩阵相对于预训练权重的偏差进行微调。该方法在11个少样本分类基准测试中，在“少样本验证”和“免验证”两种设置下均实现了最先进的性能。此外，通过ProLIP的视角重新审视非线性CLIP-Adapter，我们设计了一种正则化线性适配器（RLA），其表现更优、无需超参数调优、对学习率值不敏感，并在模型权重不可访问的黑盒场景中为ProLIP提供了替代方案。除了少样本分类，ProLIP在跨数据集迁移、领域泛化、基类到新类泛化以及测试时适应方面表现卓越——在这些任务中，其性能优于提示调优，且训练速度快一个数量级。代码发布于https://github.com/astra-vision/ProLIP。