Recently, several studies consider the stochastic optimization problem but in a heavy-tailed noise regime, i.e., the difference between the stochastic gradient and the true gradient is assumed to have a finite $p$-th moment (say being upper bounded by $\sigma^{p}$ for some $\sigma\geq0$) where $p\in(1,2]$, which not only generalizes the traditional finite variance assumption ($p=2$) but also has been observed in practice for several different tasks. Under this challenging assumption, lots of new progress has been made for either convex or nonconvex problems, however, most of which only consider smooth objectives. In contrast, people have not fully explored and well understood this problem when functions are nonsmooth. This paper aims to fill this crucial gap by providing a comprehensive analysis of stochastic nonsmooth convex optimization with heavy-tailed noises. We revisit a simple clipping-based algorithm, whereas, which is only proved to converge in expectation but under the additional strong convexity assumption. Under appropriate choices of parameters, for both convex and strongly convex functions, we not only establish the first high-probability rates but also give refined in-expectation bounds compared with existing works. Remarkably, all of our results are optimal (or nearly optimal up to logarithmic factors) with respect to the time horizon $T$ even when $T$ is unknown in advance. Additionally, we show how to make the algorithm parameter-free with respect to $\sigma$, in other words, the algorithm can still guarantee convergence without any prior knowledge of $\sigma$.

翻译：近年来，多项研究关注随机优化问题，但考虑的是重尾噪声场景，即随机梯度与真实梯度之差被假定具有有限的 $p$ 阶矩（例如，对于某个 $\sigma \geq 0$，上界为 $\sigma^{p}$），其中 $p \in (1,2]$。这不仅推广了传统的有限方差假设（$p=2$），且在多个实际任务中已被观测到。在此挑战性假设下，针对凸或非凸问题已取得诸多新进展，然而，其中大多数仅考虑光滑目标函数。相比之下，当函数非光滑时，该问题尚未被充分探索和深入理解。本文旨在通过全面分析重尾噪声下的随机非光滑凸优化来填补这一关键空白。我们重新审视一个基于剪切的简单算法，而该算法此前仅在额外强凸性假设下被证明在期望意义上收敛。在适当的参数选择下，对于凸函数和强凸函数，我们不仅首次建立了高概率收敛率，而且相比现有工作，给出了改进的期望界。值得注意的是，我们的所有结果在时间范围 $T$ 上均为最优（或达到接近最优的对数因子），即使 $T$ 事先未知。此外，我们展示了如何使算法在 $\sigma$ 方面实现免参数化，即算法无需任何关于 $\sigma$ 的先验知识即可保证收敛。