Large-scale vision-language models (VLMs) pre-trained on billion-level data have learned general visual representations and broad visual concepts. In principle, the well-learned knowledge structure of the VLMs should be inherited appropriately when being transferred to downstream tasks with limited data. However, most existing efficient transfer learning (ETL) approaches for VLMs either damage or are excessively biased towards the prior knowledge, e.g., prompt tuning (PT) discards the pre-trained text-based classifier and builds a new one while adapter-style tuning (AT) fully relies on the pre-trained features. To address this, we propose a new efficient tuning approach for VLMs named Task Residual Tuning (TaskRes), which performs directly on the text-based classifier and explicitly decouples the prior knowledge of the pre-trained models and new knowledge regarding a target task. Specifically, TaskRes keeps the original classifier weights from the VLMs frozen and obtains a new classifier for the target task by tuning a set of prior-independent parameters as a residual to the original one, which enables reliable prior knowledge preservation and flexible task-specific knowledge exploration. The proposed TaskRes is simple yet effective, which significantly outperforms previous ETL methods (e.g., PT and AT) on 11 benchmark datasets while requiring minimal effort for the implementation. Our code is available at https://github.com/geekyutao/TaskRes.

翻译：大规模预训练于十亿级数据的视觉-语言模型（VLM）已学习到通用的视觉表征及广泛的视觉概念。原则上，当这些模型被迁移到数据有限的特定下游任务时，其精心构建的知识结构应得到恰当继承。然而，现有大多数针对VLM的高效迁移学习（ETL）方法或破坏先验知识，或对其产生过度偏置——例如，提示调优（PT）丢弃了预训练的文本分类器并重新构建，而适配器式调优（AT）则完全依赖预训练特征。针对此问题，我们提出一种名为任务残差调优（TaskRes）的新型VLM高效调优方法。该方法直接作用于文本分类器，并显式解耦预训练模型的先验知识与目标任务的新知识。具体而言，TaskRes保持VLM原始分类器权重不变，通过调优一组与先验无关的参数作为原始权重的残差，为目标任务构建新分类器，从而实现可靠的先验知识保留与灵活的任务特定知识探索。所提TaskRes方法简洁高效，在11个基准数据集上显著超越先前ETL方法（例如PT和AT），且实现代价极小。我们的代码开源地址为：https://github.com/geekyutao/TaskRes。