Continual learning aims to refine model parameters for new tasks while retaining knowledge from previous tasks. Recently, prompt-based learning has emerged to leverage pre-trained models to be prompted to learn subsequent tasks without the reliance on the rehearsal buffer. Although this approach has demonstrated outstanding results, existing methods depend on preceding task-selection process to choose appropriate prompts. However, imperfectness in task-selection may lead to negative impacts on the performance particularly in the scenarios where the number of tasks is large or task distributions are imbalanced. To address this issue, we introduce I-Prompt, a task-agnostic approach focuses on the visual semantic information of image tokens to eliminate task prediction. Our method consists of semantic prompt matching, which determines prompts based on similarities between tokens, and image token-level prompting, which applies prompts directly to image tokens in the intermediate layers. Consequently, our method achieves competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. Moreover, we demonstrate the superiority of our method across various scenarios through extensive experiments.

翻译：持续学习旨在为处理新任务优化模型参数，同时保留来自先前任务的知识。近年来，基于提示的学习方法应运而生，利用预训练模型通过提示学习后续任务，无需依赖重放缓冲区。尽管该方法已展现出卓越成果，但现有方法依赖前序任务选择过程来选取合适提示。然而，任务选择的不完善可能对性能产生负面影响，尤其在任务数量庞大或任务分布不平衡的场景中。为解决这一问题，我们提出I-Prompt——一种任务无关方法，聚焦于图像令牌的视觉语义信息以消除任务预测。该方法包含语义提示匹配（通过令牌间的相似性确定提示）和图像令牌级提示（在中间层直接将提示应用于图像令牌）两部分。由此，我们的方法在四个基准测试中取得具有竞争力的性能，同时相较于现有最优方法显著缩短训练时间。此外，通过大量实验，我们在多种场景中验证了该方法的优越性。