Wall-clock convergence time and communication rounds are critical performance metrics in distributed learning with parameter-server setting. While synchronous methods converge fast but are not robust to stragglers; and asynchronous ones can reduce the wall-clock time per round but suffers from degraded convergence rate due to the staleness of gradients, it is natural to combine the two methods to achieve a balance. In this work, we develop a novel asynchronous strategy that leverages the advantages of both synchronous methods and asynchronous ones, named adaptive bounded staleness (ABS). The key enablers of ABS are two-fold. First, the number of workers that the PS waits for per round for gradient aggregation is adaptively selected to strike a straggling-staleness balance. Second, the workers with relatively high staleness are required to start a new round of computation to alleviate the negative effect of staleness. Simulation results are provided to demonstrate the superiority of ABS over state-of-the-art schemes in terms of wall-clock time and communication rounds.

翻译：摘要：在参数服务器架构下的分布式学习中，时钟收敛时间与通信轮次是关键性能指标。同步方法收敛速度快，但对慢节点缺乏鲁棒性；异步方法虽能减少每轮时钟时间，却因梯度陈旧性导致收敛速率下降。因此，结合两种方法以实现平衡是自然的思路。本文提出了一种融合同步与异步方法优势的新型异步策略——自适应有界陈旧性（ABS）。ABS的核心机制包含两点：首先，参数服务器每轮等待进行梯度聚合的工作节点数量通过自适应选择，以在慢节点容忍度与梯度陈旧性之间取得平衡；其次，要求陈旧度较高的工作节点启动新一轮计算，以减轻陈旧性的负面影响。仿真结果证明，在时钟时间与通信轮次方面，ABS相较于现有最优方案具有显著优势。