Distributed deep learning (DDL) is a promising research area, which aims to increase the efficiency of training deep learning tasks with large size of datasets and models. As the computation capability of DDL nodes continues to increase, the network connection between nodes is becoming a major bottleneck. Various methods of gradient compression and improved model synchronization have been proposed to address this bottleneck in Parameter-Server-based DDL. However, these two types of methods can result in accuracy loss due to discarded gradients and have limited enhancement on the throughput of model synchronization, respectively. To address these challenges, we propose a new model synchronization method named Overlapped Synchronization Parallel (OSP), which achieves efficient communication with a 2-stage synchronization approach and uses Local-Gradient-based Parameter correction (LGP) to avoid accuracy loss caused by stale parameters. The prototype of OSP has been implemented using PyTorch and evaluated on commonly used deep learning models and datasets with a 9-node testbed. Evaluation results show that OSP can achieve up to 50\% improvement in throughput without accuracy loss compared to popular synchronization models.

翻译：分布式深度学习（DDL）是一个有前景的研究领域，旨在提升大规模数据集和模型的深度学习任务训练效率。随着DDL节点计算能力的持续增强，节点间的网络连接逐渐成为主要瓶颈。针对基于参数服务器的DDL中的这一瓶颈，已有多种梯度压缩和改进模型同步的方法被提出。然而，这两类方法分别会因丢弃梯度而导致精度损失，或对模型同步吞吐量的提升有限。为应对这些挑战，我们提出一种新的模型同步方法——重叠同步并行（OSP），该方法通过两阶段同步实现高效通信，并采用基于局部梯度的参数校正（LGP）来避免由陈旧参数引起的精度损失。OSP原型已在PyTorch上实现，并在9节点测试平台上使用常用深度学习模型和数据集进行评估。评估结果表明，与流行的同步模型相比，OSP在不损失精度的前提下可实现高达50%的吞吐量提升。