Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. In this paper, we study how communication should be scheduled over time, including determining when and how frequently devices synchronize. Counterintuitive empirical results show that concentrating communication budgets in the later stages of decentralized training remarkably improves global test performance. Surprisingly, we uncover that fully connected communication at the final step, implemented by a single global merging, can significantly improve the performance of decentralized learning under high data heterogeneity. Our theoretical contributions, which explain these phenomena, are the first to establish that the globally merged model of decentralized SGD can match the convergence rate of parallel SGD. Technically, we reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components essential for matching this rate. This work provides evidence that decentralized learning is able to generalize under high data heterogeneity and limited communication, while offering broad new avenues for model merging research.

翻译：去中心化学习为基于参数服务器的训练提供了可扩展的替代方案，但其性能常受限于有限的点对点通信。本文研究如何调度通信时序，包括决定设备同步的时间与频率。反直觉的实验结果表明，将通信预算集中在去中心化训练后期，能显著提升全局测试性能。令人惊讶的是，我们发现在最终步骤实现的全连接通信（通过单次全局合并）能在高度数据异质性下显著提升去中心化学习性能。为解释这些现象，我们的理论贡献首次证明：去中心化SGD的全局合并模型可匹配并行SGD的收敛速率。在技术层面，我们重新解释了局部模型间的部分差异——先前被视为有害噪声的差异，实则为匹配该速率所需的结构性成分。本研究为去中心化学习在高度数据异质性与有限通信条件下的泛化能力提供了证据，同时为模型合并研究开辟了广阔新方向。