Large-scale vision-language pre-trained models have shown promising transferability to various downstream tasks. As the size of these foundation models and the number of downstream tasks grow, the standard full fine-tuning paradigm becomes unsustainable due to heavy computational and storage costs. This paper proposes UniAdapter, which unifies unimodal and multimodal adapters for parameter-efficient cross-modal adaptation on pre-trained vision-language models. Specifically, adapters are distributed to different modalities and their interactions, with the total number of tunable parameters reduced by partial weight sharing. The unified and knowledge-sharing design enables powerful cross-modal representations that can benefit various downstream tasks, requiring only 1.0%-2.0% tunable parameters of the pre-trained model. Extensive experiments on 6 cross-modal downstream benchmarks (including video-text retrieval, image-text retrieval, VideoQA, and VQA) show that in most cases, UniAdapter not only outperforms the state-of-the-arts, but even beats the full fine-tuning strategy. Particularly, on the MSRVTT retrieval task, UniAdapter achieves 49.7% recall@1 with 2.2% model parameters, outperforming the latest competitors by 2.0%. The code and models are available at https://github.com/RERV/UniAdapter.

翻译：大规模视觉-语言预训练模型已展现出对多种下游任务的强大迁移能力。随着这些基础模型规模及下游任务数量的增长，标准全微调范式因计算与存储成本过高而难以为继。本文提出UniAdapter，通过统一单模态与多模态适配器，实现预训练视觉-语言模型的参数高效跨模态适应。具体而言，适配器分布于不同模态及其交互环节，并通过部分权重共享减少可调参数总量。这种统一且知识共享的设计能够生成强大的跨模态表征，使多种下游任务受益，仅需预训练模型1.0%-2.0%的可调参数。在6个跨模态下游基准任务（包括视频-文本检索、图像-文本检索、VideoQA和VQA）上的大量实验表明：在多数情况下，UniAdapter不仅超越现有最优方法，甚至优于全微调策略。尤其在MSRVTT检索任务上，UniAdapter以2.2%的模型参数实现了49.7%的Recall@1，比最新竞争对手提升2.0%。代码与模型已发布于https://github.com/RERV/UniAdapter。