To facilitate the evolution of edge intelligence in ever-changing environments, we study on-device incremental learning constrained in limited computation resource in this paper. Current on-device training methods just focus on efficient training without considering the catastrophic forgetting, preventing the model getting stronger when continually exploring the world. To solve this problem, a direct solution is to involve the existing incremental learning mechanisms into the on-device training framework. Unfortunately, such a manner cannot work well as those mechanisms usually introduce large additional computational cost to the network optimization process, which would inevitably exceed the memory capacity of the edge devices. To address this issue, this paper makes an early effort to propose a simple but effective edge-friendly incremental learning framework. Based on an empirical study on the knowledge intensity of the kernel elements of the neural network, we find that the center kernel is the key for maximizing the knowledge intensity for learning new data, while freezing the other kernel elements would get a good balance on the model's capacity for overcoming catastrophic forgetting. Upon this finding, we further design a center-sensitive kernel optimization framework to largely alleviate the cost of the gradient computation and back-propagation. Besides, a dynamic channel element selection strategy is also proposed to facilitate a sparse orthogonal gradient projection for further reducing the optimization complexity, upon the knowledge explored from the new task data. Extensive experiments validate our method is efficient and effective, e.g., our method achieves average accuracy boost of 38.08% with even less memory and approximate computation compared to existing on-device training methods, indicating its significant potential for on-device incremental learning.

翻译：为促进边缘智能在动态环境中的演进，本文研究在有限计算资源约束下的设备端增量学习。现有设备端训练方法仅关注训练效率，未考虑灾难性遗忘问题，导致模型在持续探索世界时无法持续增强。为解决该问题，一种直接方案是将现有增量学习机制融入设备端训练框架。然而，此类机制通常会给网络优化过程引入大量额外计算开销，不可避免地超出边缘设备的存储容量。针对这一挑战，本文率先提出一种简洁高效的边缘友好型增量学习框架。基于对神经网络核元素知识密度的实证研究，我们发现中心核是最大化新数据学习知识密度的关键，而冻结其他核元素可在模型克服灾难性遗忘的能力上取得良好平衡。基于此发现，我们进一步设计中心敏感核优化框架，大幅降低梯度计算与反向传播的成本。此外，通过从新任务数据中挖掘知识，提出动态通道元素选择策略以实现稀疏正交梯度投影，从而进一步降低优化复杂度。大量实验验证了本方法的高效性：相较于现有设备端训练方法，本方法以更低内存和近似计算量实现了38.08%的平均精度提升，彰显了其在设备端增量学习领域的巨大潜力。