Recently, large-scale pre-trained Vision-Language Models (VLMs) have demonstrated great potential in learning open-world visual representations, and exhibit remarkable performance across a wide range of downstream tasks through efficient fine-tuning. In this work, we innovatively introduce the concept of dual learning into fine-tuning VLMs, i.e., we not only learn what an image is, but also what an image isn't. Building on this concept, we introduce a novel DualAdapter approach to enable dual-path adaptation of VLMs from both positive and negative perspectives with only limited annotated samples. In the inference stage, our DualAdapter performs unified predictions by simultaneously conducting complementary positive selection and negative exclusion across target classes, thereby enhancing the overall recognition accuracy of VLMs in downstream tasks. Our extensive experimental results across 15 datasets validate that the proposed DualAdapter outperforms existing state-of-the-art methods on both few-shot learning and domain generalization tasks while achieving competitive computational efficiency. Code is available at https://github.com/zhangce01/DualAdapter.

翻译：近期，大规模预训练视觉-语言模型（VLM）在学习开放世界视觉表征方面展现出巨大潜力，并通过高效微调在各类下游任务中取得了卓越性能。本文创新性地将双重学习概念引入VLM微调过程，即不仅学习图像"是什么"，还学习图像"不是什么"。基于这一理念，我们提出新型DualAdapter方法，仅需少量标注样本即可从正向与负向双视角实现VLM的双路径适配。在推理阶段，DualAdapter通过同时对目标类别执行互补的正向选择与负向排除，实现统一预测，从而提升VLM在下游任务中的整体识别准确率。我们在15个数据集上的大量实验结果表明，所提出的DualAdapter方法在少样本学习和领域泛化任务中均优于现有最先进方法，同时保持了具有竞争力的计算效率。代码开源于https://github.com/zhangce01/DualAdapter。