Distributed training of Deep Learning models has been critical to many recent successes in the field. Current standard methods primarily rely on synchronous centralized algorithms which induce major communication bottlenecks and limit their usability to High-Performance Computing (HPC) environments with strong connectivity. Decentralized asynchronous algorithms are emerging as a potential alternative but their practical applicability still lags. In this work, we focus on peerto-peer asynchronous methods due to their flexibility and parallelization potentials. In order to mitigate the increase in bandwidth they require at large scale and in poorly connected contexts, we introduce a principled asynchronous, randomized, gossip-based algorithm which works thanks to a continuous momentum named $\textbf{A}^2\textbf{CiD}^2$. In addition to inducing a significant communication acceleration at no cost other than doubling the parameters, minimal adaptation is required to incorporate $\textbf{A}^2\textbf{CiD}^2$ to other asynchronous approaches. We demonstrate its efficiency theoretically and numerically. Empirically on the ring graph, adding $\textbf{A}^2\textbf{CiD}^2$ has the same effect as doubling the communication rate. In particular, we show consistent improvement on the ImageNet dataset using up to 64 asynchronous workers (A100 GPUs) and various communication network topologies.

翻译：摘要：深度学习模型的分布式训练是近年来该领域取得诸多成功的关键因素。当前标准方法主要依赖同步集中式算法，这些算法会引发严重的通信瓶颈，并限制其在具有强连接性的高性能计算（HPC）环境中的可用性。去中心化异步算法作为潜在替代方案正逐渐兴起，但其实际适用性仍显滞后。在本工作中，我们侧重于对等异步方法，因其具有灵活性和并行化潜力。为了缓解在大规模和弱连接场景下所需带宽的增加问题，我们引入了一种基于原则的异步随机化八卦算法，该算法通过一种连续动量机制运行，命名为$\textbf{A}^2\textbf{CiD}^2$。该算法除了需要参数翻倍外无需额外成本即可显著加速通信，且将其整合到其他异步方法中仅需最小化调整。我们从理论上和数值上证明了其效率。在环形图上的实验表明，加入$\textbf{A}^2\textbf{CiD}^2$的效果等同于通信速率翻倍。特别地，我们在ImageNet数据集上使用多达64个异步工作节点（A100 GPU）及多种通信网络拓扑结构时，均观察到了性能的持续提升。