Training at the edge utilizes continuously evolving data generated at different locations. Privacy concerns prohibit the co-location of this spatially as well as temporally distributed data, deeming it crucial to design training algorithms that enable efficient continual learning over decentralized private data. Decentralized learning allows serverless training with spatially distributed data. A fundamental barrier in such distributed learning is the high bandwidth cost of communicating model updates between agents. Moreover, existing works under this training paradigm are not inherently suitable for learning a temporal sequence of tasks while retaining the previously acquired knowledge. In this work, we propose CoDeC, a novel communication-efficient decentralized continual learning algorithm which addresses these challenges. We mitigate catastrophic forgetting while learning a task sequence in a decentralized learning setup by combining orthogonal gradient projection with gossip averaging across decentralized agents. Further, CoDeC includes a novel lossless communication compression scheme based on the gradient subspaces. We express layer-wise gradients as a linear combination of the basis vectors of these gradient subspaces and communicate the associated coefficients. We theoretically analyze the convergence rate for our algorithm and demonstrate through an extensive set of experiments that CoDeC successfully learns distributed continual tasks with minimal forgetting. The proposed compression scheme results in up to 4.8x reduction in communication costs with iso-performance as the full communication baseline.

翻译：摘要：边缘训练依赖不同位置持续产生的动态数据。隐私问题禁止将这种空间与时间分布的数据集中处理，因此设计能够对分布式私有数据进行高效持续学习的训练算法至关重要。分布式学习允许利用空间分布数据实现无服务器训练。此类分布式学习的基本障碍在于智能体间传递模型更新的高带宽成本。此外，现有分布式学习范式下的研究工作本质上不适用于学习时序任务序列同时保持先前知识。本文提出CoDeC——一种新颖的高效通信分布式持续学习算法，以解决上述挑战。我们通过将正交梯度投影与分布式智能体间的高斯平均相结合，在分布式学习框架下学习任务序列时缓解灾难性遗忘。进一步，CoDeC包含基于梯度子空间的无损通信压缩方案：将逐层梯度表示为梯度子空间基向量的线性组合，并传输相应的系数。我们从理论上分析了算法的收敛速率，并通过大量实验证明CoDeC能够在遗忘最小化的前提下成功学习分布式持续任务。所提出的压缩方案相比全通信基线，在保持同等性能的情况下可将通信成本降低4.8倍。