Decentralized optimization has emerged as a critical paradigm for distributed learning, enabling scalable training while preserving data privacy through peer-to-peer collaboration. However, existing methods often suffer from communication bottlenecks due to frequent synchronization between nodes. We present Overlapping Local Decentralized SGD (OLDSGD), a novel approach to accelerate decentralized training by computation-communication overlapping, significantly reducing network idle time. With a deliberately designed update, OLDSGD preserves the same average update as Local SGD while avoiding communication-induced stalls. Theoretically, we establish non-asymptotic convergence rates for smooth non-convex objectives, showing that OLDSGD retains the same iteration complexity as standard Local Decentralized SGD while improving per-iteration runtime. Empirical results demonstrate OLDSGD's consistent improvements in wall-clock time convergence under different levels of communication delays. With minimal modifications to existing frameworks, OLDSGD offers a practical solution for faster decentralized learning without sacrificing theoretical guarantees.

翻译：去中心化优化已成为分布式学习的关键范式，通过点对点协作实现可扩展训练，同时保护数据隐私。然而，现有方法常因节点间频繁同步而面临通信瓶颈。本文提出重叠本地去中心化随机梯度下降（OLDSGD），这是一种通过计算-通信重叠加速去中心化训练的新方法，能显著减少网络空闲时间。通过精心设计的更新机制，OLDSGD在保持与本地SGD相同平均更新的同时，避免了通信引起的停滞。理论上，我们为光滑非凸目标建立了非渐近收敛速率，证明OLDSGD在保持与标准本地去中心化SGD相同迭代复杂度的同时，改善了每次迭代的运行时间。实证结果表明，在不同程度的通信延迟下，OLDSGD在挂钟时间收敛性上均能实现持续改进。通过对现有框架进行最小修改，OLDSGD为更快的去中心化学习提供了实用解决方案，且不牺牲理论保证。