This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node. More specifically, we consider a structured stochastic weakly convex objective function which is the composition of a convex function and a smooth nonconvex function. Recently, Xu et al. 2022 showed that an inertial stochastic subgradient method converges at a rate of $\mathcal{O}(\tau/\sqrt{K})$, which suffers a significant penalty from the maximum information delay $\tau$. To alleviate this issue, we propose a new delayed stochastic prox-linear ($\texttt{DSPL}$) method in which the master performs the proximal update of the parameters and the workers only need to linearly approximate the inner smooth function. Somewhat surprisingly, we show that the delays only affect the high order term in the complexity rate and hence, are negligible after a certain number of $\texttt{DSPL}$ iterations. Moreover, to further improve the empirical performance, we propose a delayed extrapolated prox-linear ($\texttt{DSEPL}$) method which employs Polyak-type momentum to speed up the algorithm convergence. Building on the tools for analyzing $\texttt{DSPL}$, we also develop improved analysis of delayed stochastic subgradient method ($\texttt{DSGD}$). In particular, for general weakly convex problems, we show that convergence of $\texttt{DSGD}$ only depends on the expected delay.

翻译：本文研究了在含有连接至主节点的工人节点的分布式网络中，针对弱凸优化的延迟随机算法。更具体地，我们考虑一个结构化的随机弱凸目标函数，该函数由凸函数与光滑非凸函数复合而成。近期，Xu等人2022年研究表明，惯性随机次梯度方法以$\mathcal{O}(\tau/\sqrt{K})$速率收敛，但会受到最大信息延迟$\tau$的显著惩罚。为缓解此问题，我们提出一种新型延迟随机近端线性算法，其中主节点执行参数近端更新，而工人节点仅需线性逼近内部光滑函数。令人意外的是，我们证明延迟仅影响复杂度速率的高阶项，因此在执行一定数量的迭代后延迟可忽略不计。此外，为进一步提升经验性能，我们提出延迟外推近端线性算法，该方法采用Polyak型动量加速算法收敛。基于分析DSPL的工具，我们还对延迟随机次梯度法开发了改进的分析。特别地，对于一般弱凸问题，我们证明DSGD的收敛性仅依赖于期望延迟。