Distributed learning is essential to train machine learning algorithms across heterogeneous agents while maintaining data privacy. We conduct an asymptotic analysis of Unified Distributed SGD (UD-SGD), exploring a variety of communication patterns, including decentralized SGD and local SGD within Federated Learning (FL), as well as the increasing communication interval in the FL setting. In this study, we assess how different sampling strategies, such as i.i.d. sampling, shuffling, and Markovian sampling, affect the convergence speed of UD-SGD by considering the impact of agent dynamics on the limiting covariance matrix as described in the Central Limit Theorem (CLT). Our findings not only support existing theories on linear speedup and asymptotic network independence, but also theoretically and empirically show how efficient sampling strategies employed by individual agents contribute to overall convergence in UD-SGD. Simulations reveal that a few agents using highly efficient sampling can achieve or surpass the performance of the majority employing moderately improved strategies, providing new insights beyond traditional analyses focusing on the worst-performing agent.

翻译：分布式学习对于在保持数据隐私的同时跨异构智能体训练机器学习算法至关重要。本文对统一分布式随机梯度下降（UD-SGD）进行了渐近分析，探讨了多种通信模式，包括去中心化SGD、联邦学习（FL）中的本地SGD，以及FL设置中不断增加的通信间隔。本研究通过分析智能体动态对中心极限定理（CLT）所描述的极限协方差矩阵的影响，评估了不同采样策略（如独立同分布采样、混洗采样和马尔可夫采样）如何影响UD-SGD的收敛速度。我们的研究结果不仅支持了关于线性加速和渐近网络独立性的现有理论，还从理论和实证两方面揭示了单个智能体采用的高效采样策略如何促进UD-SGD的整体收敛。仿真实验表明，少数采用高效采样策略的智能体能够达到甚至超越多数采用中等改进策略的智能体的性能，这为超越传统聚焦最差性能智能体的分析提供了新的见解。