Decentralized learning enables the training of deep learning models over large distributed datasets generated at different locations, without the need for a central server. However, in practical scenarios, the data distribution across these devices can be significantly different, leading to a degradation in model performance. In this paper, we focus on designing a decentralized learning algorithm that is less susceptible to variations in data distribution across devices. We propose Global Update Tracking (GUT), a novel tracking-based method that aims to mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a $1-6\%$ improvement in test accuracy compared to other existing techniques.

翻译：去中心化学习能够在不依赖中央服务器的情况下，基于不同位置生成的大规模分布式数据集训练深度学习模型。然而，在实际场景中，这些设备上的数据分布可能存在显著差异，导致模型性能下降。本文聚焦于设计一种对设备间数据分布差异不敏感的去中心化学习算法。我们提出了一种新颖的基于追踪的方法——全局更新追踪（GUT），旨在缓解去中心化学习中异构数据的影响，且无需引入任何通信开销。通过在多个计算机视觉数据集（CIFAR-10、CIFAR-100、Fashion MNIST和ImageNette）、多种模型架构及网络拓扑结构上进行的全面实验，我们验证了所提技术的有效性。实验结果表明，与现有其他技术相比，所提方法在异构数据下的去中心化学习中实现了最优性能，测试准确率提升了$1-6\%$。