Continual learning can empower vision-language models to continuously acquire new knowledge, without the need for access to the entire historical dataset. However, mitigating the performance degradation in large-scale models is non-trivial due to (i) parameter shifts throughout lifelong learning and (ii) significant computational burdens associated with full-model tuning. In this work, we present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters in response to new tasks. To preserve the zero-shot recognition capability of vision-language models, we further introduce a Distribution Discriminative Auto-Selector (DDAS) that automatically routes in-distribution and out-of-distribution inputs to the MoE Adapter and the original CLIP, respectively. Through extensive experiments across various settings, our proposed method consistently outperforms previous state-of-the-art approaches while concurrently reducing parameter training burdens by 60%. Our code locates at https://github.com/JiazuoYu/MoE-Adapters4CL

翻译：持续学习能够使视觉-语言模型在不依赖完整历史数据集的情况下持续获取新知识。然而，缓解大规模模型性能退化并非易事，原因在于：(i) 终身学习过程中的参数偏移，以及(ii) 全模型微调带来的显著计算负担。本文提出一种参数高效的持续学习框架，以缓解视觉-语言模型增量学习中的长期遗忘问题。该方法通过集成面向新任务的混合专家（MoE）适配器，动态扩展预训练CLIP模型。为保持视觉-语言模型的零样本识别能力，我们进一步引入分布判别自动选择器（DDAS），该选择器可自动将分布内输入路由至MoE适配器处理，将分布外输入路由至原始CLIP模型处理。在多种设置下的广泛实验表明，所提方法不仅持续超越现有最先进方法，同时将参数训练负担降低60%。我们的代码位于https://github.com/JiazuoYu/MoE-Adapters4CL