Large amounts of incremental learning algorithms have been proposed to alleviate the catastrophic forgetting issue arises while dealing with sequential data on a time series. However, the adversarial robustness of incremental learners has not been widely verified, leaving potential security risks. Specifically, for poisoning-based backdoor attacks, we argue that the nature of streaming data in IL provides great convenience to the adversary by creating the possibility of distributed and cross-task attacks -- an adversary can affect \textbf{any unknown} previous or subsequent task by data poisoning \textbf{at any time or time series} with extremely small amount of backdoor samples injected (e.g., $0.1\%$ based on our observations). To attract the attention of the research community, in this paper, we empirically reveal the high vulnerability of 11 typical incremental learners against poisoning-based backdoor attack on 3 learning scenarios, especially the cross-task generalization effect of backdoor knowledge, while the poison ratios range from $5\%$ to as low as $0.1\%$. Finally, the defense mechanism based on activation clustering is found to be effective in detecting our trigger pattern to mitigate potential security risks.

翻译：大量增量学习算法已被提出，以缓解处理时序序列数据时出现的灾难性遗忘问题。然而，增量学习者的对抗鲁棒性尚未得到广泛验证，存在潜在安全风险。具体而言，对于基于投毒的后门攻击，我们认为增量学习中流式数据的特性为攻击者提供了极大便利，通过创建分布式和跨任务攻击的可能性——攻击者可以在**任意时间或时间序列**中，通过注入极少量的后门样本（例如，基于我们的观察，仅需$0.1\%$）来影响**任何未知**的前序或后续任务。为引起研究社区的关注，本文实证揭示了11种典型增量学习者在3种学习场景下对基于投毒的后门攻击的高度脆弱性，尤其是后门知识的跨任务泛化效应，其中投毒比例从$5\%$低至$0.1\%$。最后，发现基于激活聚类的防御机制能有效检测我们的触发模式，从而缓解潜在安全风险。