Continual learning (CL) aims to help deep neural networks to learn new knowledge while retaining what has been learned. Recently, pre-trained vision-language models such as CLIP, with powerful generalizability, have been gaining traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks calls for finetuning of the CLIP on the latter. The deterministic nature of the existing finetuning methods makes them overlook the many possible interactions across the modalities and deems them unsafe for high-risk CL tasks requiring reliable uncertainty estimation. To address these, our work proposes Continual LeArning with Probabilistic finetuning (CLAP). CLAP develops probabilistic modeling over task-specific modules with visual-guided text features, providing more calibrated finetuning in CL. It further alleviates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific modules. Cooperating with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-box applications of superior uncertainty estimation abilities of CLAP for novel data detection and exemplar selection within CL setups. Our code is available at \url{https://github.com/srvCodes/clap4clip}.

翻译：持续学习（CL）旨在帮助深度神经网络在学习新知识的同时保留已学内容。近年来，具有强大泛化能力的预训练视觉语言模型（如CLIP）正逐渐成为实用的持续学习候选方案。然而，预训练任务与下游持续学习任务之间的领域不匹配要求对CLIP进行微调。现有微调方法的确定性特性使其忽视了跨模态的多种潜在交互，且对于需要可靠不确定性估计的高风险持续学习任务而言不够安全。为解决这些问题，本文提出概率微调持续学习（CLAP）方法。CLAP通过视觉引导的文本特征对任务特定模块建立概率建模，为持续学习提供更校准的微调。该方法进一步利用CLIP丰富的预训练知识对任务特定模块进行权重初始化与分布正则化，从而有效缓解遗忘问题。结合多样化的现有提示方法，CLAP能够超越当前主流的确定性微调方案，实现基于CLIP的持续学习。最后，我们展示了CLAP卓越的不确定性估计能力在持续学习框架中可用于新颖数据检测和样本选择的即插即用应用。代码已开源：\url{https://github.com/srvCodes/clap4clip}。