Modern deep learning models, growing larger and more complex, have demonstrated exceptional generalization and accuracy due to training on huge datasets. This trend is expected to continue. However, the increasing size of these models poses challenges in training, as traditional centralized methods are limited by memory constraints at such scales. This paper proposes an asynchronous decentralized training paradigm for large modern deep learning models that harnesses the compute power of regular heterogeneous PCs with limited resources connected across the internet to achieve favourable performance metrics. Ravnest facilitates decentralized training by efficiently organizing compute nodes into clusters with similar data transfer rates and compute capabilities, without necessitating that each node hosts the entire model. These clusters engage in $\textit{Zero-Bubble Asynchronous Model Parallel}$ training, and a $\textit{Parallel Multi-Ring All-Reduce}$ method is employed to effectively execute global parameter averaging across all clusters. We have framed our asynchronous SGD loss function as a block structured optimization problem with delayed updates and derived an optimal convergence rate of $O\left(\frac{1}{\sqrt{K}}\right)$. We further discuss linear speedup with respect to the number of participating clusters and the bound on the staleness parameter.

翻译：现代深度学习模型日益庞大和复杂，通过在大型数据集上的训练展现出卓越的泛化能力和准确度。这一趋势预计将持续。然而，模型规模的不断增长给训练带来了挑战，因为传统的集中式训练方法在此等规模下受限于内存约束。本文提出一种针对大型现代深度学习模型的异步去中心化训练范式，该范式利用通过互联网连接的、资源有限的常规异构PC的计算能力，以实现优越的性能指标。Ravnest通过将计算节点高效组织成具有相似数据传输速率和计算能力的集群，从而促进去中心化训练，而无需每个节点都托管整个模型。这些集群参与 $\textit{Zero-Bubble Asynchronous Model Parallel}$ 训练，并采用 $\textit{Parallel Multi-Ring All-Reduce}$ 方法在所有集群间高效执行全局参数平均。我们将异步SGD损失函数构建为一个具有延迟更新的块结构优化问题，并推导出 $O\left(\frac{1}{\sqrt{K}}\right)$ 的最优收敛速率。我们进一步讨论了相对于参与集群数量的线性加速比以及陈旧度参数的边界。