We consider the distributed stochastic optimization problem where $n$ agents want to minimize a global function given by the sum of agents' local functions, and focus on the heterogeneous setting when agents' local functions are defined over non-i.i.d. data sets. We study the Local SGD method, where agents perform a number of local stochastic gradient steps and occasionally communicate with a central node to improve their local optimization tasks. We analyze the effect of local steps on the convergence rate and the communication complexity of Local SGD. In particular, instead of assuming a fixed number of local steps across all communication rounds, we allow the number of local steps during the $i$-th communication round, $H_i$, to be different and arbitrary numbers. Our main contribution is to characterize the convergence rate of Local SGD as a function of $\{H_i\}_{i=1}^R$ under various settings of strongly convex, convex, and nonconvex local functions, where $R$ is the total number of communication rounds. Based on this characterization, we provide sufficient conditions on the sequence $\{H_i\}_{i=1}^R$ such that Local SGD can achieve linear speed-up with respect to the number of workers. Furthermore, we propose a new communication strategy with increasing local steps superior to existing communication strategies for strongly convex local functions. On the other hand, for convex and nonconvex local functions, we argue that fixed local steps are the best communication strategy for Local SGD and recover state-of-the-art convergence rate results. Finally, we justify our theoretical results through extensive numerical experiments.

翻译：我们考虑了分布式的随机优化问题,即美元代理商希望将当地功能加在一起产生的全球功能降到最低,并关注在非i.i.d.数据集上界定当地代理商的当地功能时的多样化设置。我们研究了当地SGD方法,即当地代理商执行若干当地随机梯度梯度步骤,有时与中央节点通信,以改进当地优化任务。我们分析了当地步骤对当地SGD的趋同率和通信复杂性的影响。特别是,我们没有在所有通信周期中固定地采取一定数量的当地步骤,而是允许美元(H)美元(i)美元)第一轮通信周期期间当地步骤的数量不同和任意数字。我们的主要贡献是将当地SGD的趋同率定性为$H_i=i=i=1R$的功能,在不同情况下,当地通信速度和当地通信的总数是$(R$)的总数。基于这一特征,我们为美元(H)ix美元(i)当地通信周期中的固定步骤提供了足够的条件,我们为当前SGDG值战略的升级速度提供了一定速度。