Continual learning requires models to learn continuously while preserving prior knowledge under evolving data streams. Distillation-based methods are appealing for retaining past knowledge in a shared single-model framework with low storage overhead. However, they remain constrained by the stability-plasticity dilemma: knowledge acquisition and preservation are still optimized through coupled objectives, and existing enhancement methods do not alter this underlying bottleneck. To address this issue, we propose a plugin extension paradigm termed Distillation-aware Lightweight Components (DLC) for distillation-based CL. DLC deploys lightweight residual plugins into the base feature extractor's classifier-proximal layer, enabling semantic-level residual correction for better classification accuracy while minimizing disruption to the overall feature extraction process. During inference, plugin-enhanced representations are aggregated to produce classification predictions. To mitigate interference from non-target plugins, we further introduce a lightweight weighting unit that learns to assign importance scores to different plugin-enhanced representations. DLC could deliver a significant 8% accuracy gain on large-scale benchmarks while introducing only a 4% increase in backbone parameters, highlighting its exceptional efficiency. Moreover, DLC is compatible with other plug-and-play CL enhancements and delivers additional gains when combined with them.

翻译：持续学习要求模型在学习过程中保持先验知识，以应对不断演化的数据流。蒸馏方法因其在共享单模型框架中保留过去知识且存储开销低而备受青睐。然而，它们仍受制于稳定性-可塑性困境：知识获取与保留通过耦合目标优化，现有增强方法未改变这一根本瓶颈。为解决该问题，我们提出一种称为蒸馏感知轻量级组件（DLC）的插件扩展范式，用于基于蒸馏的持续学习。DLC在基础特征提取器的分类器邻近层中部署轻量级残差插件，实现语义级残差校正以提升分类精度，同时最小化对整体特征提取过程的干扰。推理时，聚合插件增强表示以生成分类预测。为缓解非目标插件的干扰，我们进一步引入一个轻量级加权单元，学习为不同插件增强表示分配重要性分数。DLC在大规模基准测试中带来8%的显著精度提升，同时参数量仅增加4%，凸显其卓越效率。此外，DLC与其他即插即用持续学习增强方法兼容，并能在结合使用时提供额外增益。