Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous SGD and its robust variant, called $m$-Synchronous SGD, and theoretically show that they are nearly optimal in many heterogeneous computation scenarios, which is somewhat unexpected. We analyze the synchronous methods under random computation times and adversarial partial participation of workers, and prove that their time complexities are optimal in many practical regimes, up to logarithmic factors. While synchronous methods are not universal solutions and there exist tasks where asynchronous methods may be necessary, we show that they are sufficient for many modern heterogeneous computation scenarios.

翻译：尽管异步优化领域近期取得了显著进展，现代分布式优化方法仍主要依赖传统的同步范式。本文重新审视同步随机梯度下降法及其鲁棒变体——称为$m$-同步随机梯度下降法，并从理论上证明它们在许多异构计算场景中具有近乎最优的性能，这一结论有些出人意料。我们在随机计算时间与对抗性部分工作节点参与的设定下分析同步方法，证明其时间复杂度在许多实际运行区间内达到对数因子意义下的最优。虽然同步方法并非通用解决方案，且存在异步方法可能必需的任务场景，但我们证明同步方法足以应对许多现代异构计算场景。