This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node. More specifically, we consider a structured stochastic weakly convex objective function which is the composition of a convex function and a smooth nonconvex function. Recently, Xu et al. 2022 showed that an inertial stochastic subgradient method converges at a rate of $\mathcal{O}(\tau/\sqrt{K})$, which suffers a significant penalty from the maximum information delay $\tau$. To alleviate this issue, we propose a new delayed stochastic prox-linear ($\texttt{DSPL}$) method in which the master performs the proximal update of the parameters and the workers only need to linearly approximate the inner smooth function. Somewhat surprisingly, we show that the delays only affect the high order term in the complexity rate and hence, are negligible after a certain number of $\texttt{DSPL}$ iterations. Moreover, to further improve the empirical performance, we propose a delayed extrapolated prox-linear ($\texttt{DSEPL}$) method which employs Polyak-type momentum to speed up the algorithm convergence. Building on the tools for analyzing $\texttt{DSPL}$, we also develop improved analysis of delayed stochastic subgradient method ($\texttt{DSGD}$). In particular, for general weakly convex problems, we show that convergence of $\texttt{DSGD}$ only depends on the expected delay.

翻译：本文研究了在分布式网络中，当工作节点连接至主节点时，用于弱凸优化的延迟随机算法。具体而言，我们考虑一个结构化随机弱凸目标函数，它由一个凸函数和一个光滑非凸函数复合而成。近期，Xu等人（2022）证明惯性随机次梯度方法以$\mathcal{O}(\tau/\sqrt{K})$的速率收敛，该方法因最大信息延迟$\tau$而遭受显著性能损失。为解决此问题，我们提出一种新的延迟随机近端线性方法（$\texttt{DSPL}$），其中主节点执行参数的近端更新，而工作节点仅需线性逼近内部光滑函数。出乎意料的是，我们证明延迟仅影响复杂度率中的高阶项，因此在经过一定数量的$\texttt{DSPL}$迭代后，延迟可被忽略。此外，为进一步提升实际性能，我们提出一种延迟外推近端线性方法（$\texttt{DSEPL}$），该方法采用Polyak型动量加速算法收敛。基于分析$\texttt{DSPL}$的工具，我们还改进了对延迟随机次梯度方法（$\texttt{DSGD}$）的分析。特别地，对于一般弱凸问题，我们证明$\texttt{DSGD}$的收敛性仅依赖于期望延迟。