Decentralized learning (DL) systems have been gaining popularity because they avoid raw data sharing by communicating only model parameters, hence preserving data confidentiality. However, the large size of deep neural networks poses a significant challenge for decentralized training, since each node needs to exchange gigabytes of data, overloading the network. In this paper, we address this challenge with JWINS, a communication-efficient and fully decentralized learning system that shares only a subset of parameters through sparsification. JWINS uses wavelet transform to limit the information loss due to sparsification and a randomized communication cut-off that reduces communication usage without damaging the performance of trained models. We demonstrate empirically with 96 DL nodes on non-IID datasets that JWINS can achieve similar accuracies to full-sharing DL while sending up to 64% fewer bytes. Additionally, on low communication budgets, JWINS outperforms the state-of-the-art communication-efficient DL algorithm CHOCO-SGD by up to 4x in terms of network savings and time.

翻译：去中心化学习系统因仅通过共享模型参数避免原始数据泄露，从而保障数据机密性而日益普及。然而，深度神经网络的庞大体积对去中心化训练构成了重大挑战——每个节点需交换数GB数据，导致网络过载。本文提出JWINS——一种通信高效且完全去中心化的学习系统，通过稀疏化仅共享参数子集以应对该挑战。JWINS利用小波变换限制稀疏化带来的信息损失，并采用随机通信中断机制，在无需降低训练模型性能的前提下减少通信开销。基于96个去中心化学习节点在非独立同分布数据集上的实验表明：JWINS能在保持与全量共享去中心化学习相近准确率的同时，最高减少64%的数据传输量。此外，在低通信预算场景下，JWINS在网络节省效率和时间性能上较当前最优的通信高效去中心化学习算法CHOCO-SGD提升高达4倍。