This paper presents fault-tolerant asynchronous Stochastic Gradient Descent (SGD) algorithms. SGD is widely used for approximating the minimum of a cost function $Q$, as a core part of optimization and learning algorithms. Our algorithms are designed for the cluster-based model, which combines message-passing and shared-memory communication layers. Processes may fail by crashing, and the algorithm inside each cluster is wait-free, using only reads and writes. For a strongly convex function $Q$, our algorithm tolerates any number of failures, and provides convergence rate that yields the maximal distributed acceleration over the optimal convergence rate of sequential SGD. For arbitrary functions, the convergence rate has an additional term that depends on the maximal difference between the parameters at the same iteration. (This holds under standard assumptions on $Q$.) In this case, the algorithm obtains the same convergence rate as sequential SGD, up to a logarithmic factor. This is achieved by using, at each iteration, a multidimensional approximate agreement algorithm, tailored for the cluster-based model. The algorithm for arbitrary functions requires that at least a majority of the clusters contain at least one nonfaulty process. We prove that this condition is necessary when optimizing some non-convex functions.

翻译：本文提出具有容错能力的异步随机梯度下降（SGD）算法。SGD广泛应用于逼近代价函数$Q$的最小值，是优化与学习算法的核心组成部分。我们的算法专为集群模型设计，该模型融合了消息传递与共享内存两种通信层。进程可能因崩溃而失效，且每个集群内部的算法采用无等待机制，仅通过读写操作实现。对于强凸函数$Q$，本算法可容忍任意数量的故障，其收敛速度可实现相较于最优串行SGD收敛速度的最大分布式加速。对于一般函数，收敛速度中增加了一项与同一迭代步参数最大差异相关的附加项（该结论基于$Q$的标准假设）。在此情况下，算法的收敛速度与串行SGD相当（仅相差对数因子）。这一成果得益于每次迭代中采用专为集群模型定制的多维近似一致性算法。处理一般函数时，要求至少半数集群包含至少一个无故障进程。我们证明，在优化某些非凸函数时，此条件具有必要性。