Continual learning on edge devices poses unique challenges due to stringent resource constraints. This paper introduces a novel method that leverages stochastic competition principles to promote sparsity, significantly reducing deep network memory footprint and computational demand. Specifically, we propose deep networks that comprise blocks of units that compete locally to win the representation of each arising new task; competition takes place in a stochastic manner. This type of network organization results in sparse task-specific representations from each network layer; the sparsity pattern is obtained during training and is different among tasks. Crucially, our method sparsifies both the weights and the weight gradients, thus facilitating training on edge devices. This is performed on the grounds of winning probability for each unit in a block. During inference, the network retains only the winning unit and zeroes-out all weights pertaining to non-winning units for the task at hand. Thus, our approach is specifically tailored for deployment on edge devices, providing an efficient and scalable solution for continual learning in resource-limited environments.

翻译：边缘设备上的持续学习由于严格的资源限制而面临独特挑战。本文提出一种新颖方法，利用随机竞争原理促进稀疏性，显著降低深度网络的内存占用和计算需求。具体而言，我们提出一种深度网络结构，其包含多个单元块，这些单元通过局部随机竞争来获得每个新任务的表征权。这种网络组织形式使得每个网络层产生稀疏的任务特定表征，且不同任务的稀疏模式在训练过程中形成并互不相同。关键的是，我们的方法同时稀疏化权重和权重梯度，从而便于在边缘设备上进行训练。这是通过计算每个块中单元的获胜概率实现的。在推理阶段，网络仅保留获胜单元，并将当前任务相关的非获胜单元所有权重置零。因此，我们的方法专门针对边缘设备部署设计，为资源受限环境中的持续学习提供了高效且可扩展的解决方案。