Optimization under heavy-tailed noise has become popular recently, since it better fits many modern machine learning tasks, as captured by empirical observations. Concretely, instead of a finite second moment on gradient noise, a bounded ${\frak p}$-th moment where ${\frak p}\in(1,2]$ has been recognized to be more realistic (say being upper bounded by $σ_{\frak l}^{\frak p}$ for some $σ_{\frak l}\ge0$). A simple yet effective operation, gradient clipping, is known to handle this new challenge successfully. Specifically, Clipped Stochastic Gradient Descent (Clipped SGD) guarantees a high-probability rate ${\cal O}(σ_{\frak l}\ln(1/δ)T^{1/{\frak p}-1})$ (resp. ${\cal O}(σ_{\frak l}^2\ln^2(1/δ)T^{2/{\frak p}-2})$) for nonsmooth convex (resp. strongly convex) problems, where $δ\in(0,1]$ is the failure probability and $T\in\mathbb{N}$ is the time horizon. In this work, we provide a refined analysis for Clipped SGD and offer two faster rates, ${\cal O}(σ_{\frak l}d_{\rm eff}^{-1/2{\frak p}}\ln^{1-1/{\frak p}}(1/δ)T^{1/{\frak p}-1})$ and ${\cal O}(σ_{\frak l}^2d_{\rm eff}^{-1/{\frak p}}\ln^{2-2/{\frak p}}(1/δ)T^{2/{\frak p}-2})$, than the aforementioned best results, where $d_{\rm eff}\ge1$ is a quantity we call the $\textit{generalized effective dimension}$. Our analysis improves upon the existing approach on two sides: better utilization of Freedman's inequality and finer bounds for clipping error under heavy-tailed noise. In addition, we extend the refined analysis to convergence in expectation and obtain new rates that break the known lower bounds. Lastly, to complement the study, we establish new lower bounds for both high-probability and in-expectation convergence. Notably, the in-expectation lower bounds match our new upper bounds, indicating the optimality of our refined analysis for convergence in expectation.

翻译：重尾噪声下的优化问题近来备受关注，因其更贴合众多现代机器学习任务的经验观测特征。具体而言，梯度噪声不再具有有限二阶矩，而是满足有界${\frak p}$阶矩（${\frak p}\in(1,2]$），即存在$σ_{\frak l}\ge0$使得${\rm E}\|\nabla \cdot\|^{\frak p}\le σ_{\frak l}^{\frak p}$。梯度裁剪这一简洁高效的操作已被证明能成功应对这一新挑战。具体地，裁剪随机梯度下降法（Clipped SGD）在非光滑凸（强凸）问题上可实现高概率收敛速率${\cal O}(σ_{\frak l}\ln(1/δ)T^{1/{\frak p}-1})$（对应${\cal O}(σ_{\frak l}^2\ln^2(1/δ)T^{2/{\frak p}-2})$），其中$δ\in(0,1]$为失败概率，$T\in\mathbb{N}$为时间步长。本文对Clipped SGD进行精细分析，提出两个更优收敛速率：${\cal O}(σ_{\frak l}d_{\rm eff}^{-1/2{\frak p}}\ln^{1-1/{\frak p}}(1/δ)T^{1/{\frak p}-1})$和${\cal O}(σ_{\frak l}^2d_{\rm eff}^{-1/{\frak p}}\ln^{2-2/{\frak p}}(1/δ)T^{2/{\frak p}-2})$，优于现有最优结果，其中$d_{\rm eff}\ge1$为本文定义的$\textit{广义有效维度}$。我们的分析从两方面改进了现有方法：更高效利用Freedman不等式，以及更精细刻画重尾噪声下的裁剪误差。此外，我们将精细分析拓展至期望收敛情形，获得突破已知下界的收敛速率。最后，作为补充研究，我们建立了高概率收敛与期望收敛的新下界。值得注意的是，期望收敛下界与本文新上界匹配，表明期望收敛精细分析的最优性。