We study the problem of solving strongly convex and smooth unconstrained optimization problems using stochastic first-order algorithms. We devise a novel algorithm, referred to as \emph{Recursive One-Over-T SGD} (\ROOTSGD), based on an easily implementable, recursive averaging of past stochastic gradients. We prove that it simultaneously achieves state-of-the-art performance in both a finite-sample, nonasymptotic sense and an asymptotic sense. On the nonasymptotic side, we prove risk bounds on the last iterate of \ROOTSGD with leading-order terms that match the optimal statistical risk with a unity pre-factor, along with a higher-order term that scales at the sharp rate of $O(n^{-3/2})$ under the Lipschitz condition on the Hessian matrix. On the asymptotic side, we show that when a mild, one-point Hessian continuity condition is imposed, the rescaled last iterate of (multi-epoch) \ROOTSGD converges asymptotically to a Gaussian limit with the Cram\'{e}r-Rao optimal asymptotic covariance, for a broad range of step-size choices.

翻译：我们研究使用随机一阶算法求解强凸且光滑的无约束优化问题。我们提出了一种新算法，称为 \emph{递归一-over-T SGD} (\ROOTSGD)，该算法基于易于实现的过去随机梯度递归平均方法。我们证明该算法在有限样本非渐近意义和渐近意义上同时实现了最先进的性能。在非渐近方面，我们证明了 \ROOTSGD 最后迭代的风险界，其主阶项匹配了具有单位预因子的最优统计风险，而高阶项在 Hessian 矩阵 Lipschitz 条件下以 $O(n^{-3/2})$ 的锐利率缩放。在渐近方面，我们表明，当施加温和的单点 Hessian 连续性条件时，（多轮次）\ROOTSGD 重标度后的最后迭代在广泛步长选择下渐近收敛到具有 Cramér-Rao 最优渐近协方差的高斯极限。