Visual foundation models like CLIP excel in learning feature representations from extensive datasets through self-supervised methods, demonstrating remarkable transfer learning and generalization capabilities. A growing number of applications based on visual foundation models are emerging, including innovative solutions such as BLIP-2. These applications employ pre-trained CLIP models as upstream feature extractors and train various downstream modules to accomplish diverse tasks. In situations involving system upgrades that require updating the upstream foundation model, it becomes essential to re-train all downstream modules to adapt to the new foundation model, which is inflexible and inefficient. In this paper, we introduce a parameter-efficient and task-agnostic adapter, dubbed TaCA, that facilitates compatibility across distinct foundation models while ensuring enhanced performance for the new models. TaCA allows downstream applications to seamlessly integrate better-performing foundation models without necessitating retraining. We conduct extensive experimental validation of TaCA using different scales of models with up to one billion parameters on various tasks such as video-text retrieval, video recognition, and visual question answering. The results consistently demonstrate the emergent ability of TaCA on hot-plugging upgrades for visual foundation models. Codes and models will be available at https://github.com/TencentARC/TaCA.

翻译：视觉基础模型（如CLIP）通过自监督方法从大规模数据集中学习特征表示，展现出卓越的迁移学习和泛化能力。基于视觉基础模型的应用日益增多，包括BLIP-2等创新方案。这些应用采用预训练的CLIP模型作为上游特征提取器，并训练各类下游模块以完成多样化任务。当涉及系统升级需要更新上游基础模型时，所有下游模块必须重新训练以适应新模型，这种做法缺乏灵活性和效率。本文提出一种参数高效且任务无关的适配器（TaCA），该适配器在保证新模型性能提升的同时，促进不同基础模型之间的兼容性。TaCA使下游应用能够无缝集成性能更优的基础模型，而无需重新训练。我们使用参数规模达十亿的不同尺度模型，在视频-文本检索、视频识别和视觉问答等多项任务上对TaCA进行了广泛实验验证。结果一致表明TaCA在视觉基础模型热插拔升级方面的涌现能力。代码和模型将发布在https://github.com/TencentARC/TaCA。