This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node. More specifically, we consider a structured stochastic weakly convex objective function which is the composition of a convex function and a smooth nonconvex function. Recently, Xu et al. 2022 showed that an inertial stochastic subgradient method converges at a rate of $\mathcal{O}(\tau/\sqrt{K})$, which suffers a significant penalty from the maximum information delay $\tau$. To alleviate this issue, we propose a new delayed stochastic prox-linear ($\texttt{DSPL}$) method in which the master performs the proximal update of the parameters and the workers only need to linearly approximate the inner smooth function. Somewhat surprisingly, we show that the delays only affect the high order term in the complexity rate and hence, are negligible after a certain number of $\texttt{DSPL}$ iterations. Moreover, to further improve the empirical performance, we propose a delayed extrapolated prox-linear ($\texttt{DSEPL}$) method which employs Polyak-type momentum to speed up the algorithm convergence. Building on the tools for analyzing $\texttt{DSPL}$, we also develop improved analysis of delayed stochastic subgradient method ($\texttt{DSGD}$). In particular, for general weakly convex problems, we show that convergence of $\texttt{DSGD}$ only depends on the expected delay.

翻译：本文研究分布式网络中面向弱凸优化的延迟随机算法，该网络由连接至主节点的多个工作节点构成。具体而言，我们考虑一类结构化的随机弱凸目标函数，它由凸函数与光滑非凸函数复合而成。Xu等人（2022）近期证明惯性随机次梯度方法的收敛率为$\mathcal{O}(\tau/\sqrt{K})$，其性能严重受限于最大信息延迟$\tau$。为缓解该问题，我们提出一种新型延迟随机近端线性方法（$\texttt{DSPL}$），其中主节点执行参数近端更新，工作节点仅需线性逼近内部光滑函数。令人意外的是，我们证明延迟仅影响复杂度速率中的高阶项，因此在经过一定次数的$\texttt{DSPL}$迭代后可忽略不计。此外，为提升实际表现，我们进一步提出延迟外推近端线性方法（$\texttt{DSEPL}$），通过引入Polyak型动量加速算法收敛。基于$\texttt{DSPL}$分析工具，我们还改进了延迟随机次梯度方法（$\texttt{DSGD}$）的理论分析：对于一般弱凸问题，证明$\texttt{DSGD}$的收敛性仅依赖于期望延迟。