Classical convergence analyses for optimization algorithms rely on the widely-adopted uniform smoothness assumption. However, recent experimental studies have demonstrated that many machine learning problems exhibit non-uniform smoothness, meaning the smoothness factor is a function of the model parameter instead of a universal constant. In particular, it has been observed that the smoothness grows with respect to the gradient norm along the training trajectory. Motivated by this phenomenon, the recently introduced $(L_0, L_1)$-smoothness is a more general notion, compared to traditional $L$-smoothness, that captures such positive relationship between smoothness and gradient norm. Under this type of non-uniform smoothness, existing literature has designed stochastic first-order algorithms by utilizing gradient clipping techniques to obtain the optimal $\mathcal{O}(\epsilon^{-3})$ sample complexity for finding an $\epsilon$-approximate first-order stationary solution. Nevertheless, the studies of quasi-Newton methods are still lacking. Considering higher accuracy and more robustness for quasi-Newton methods, in this paper we propose a fast stochastic quasi-Newton method when there exists non-uniformity in smoothness. Leveraging gradient clipping and variance reduction, our algorithm can achieve the best-known $\mathcal{O}(\epsilon^{-3})$ sample complexity and enjoys convergence speedup with simple hyperparameter tuning. Our numerical experiments show that our proposed algorithm outperforms the state-of-the-art approaches.

翻译：经典优化算法的收敛性分析通常依赖于广泛采用的均匀光滑假设。然而，近期实验研究表明，许多机器学习问题呈现出非均匀光滑性，即光滑因子是模型参数的函数而非通用常数。具体而言，沿训练轨迹观察发现，光滑性会随梯度范数增大而增强。受此现象启发，与传统的L-光滑性相比，新近提出的(L₀, L₁)-光滑性是一种更普适的概念，能够刻画光滑性与梯度范数之间的正相关关系。在非均匀光滑性条件下，现有文献通过引入梯度裁剪技术设计了随机一阶算法，在寻找ε-近似一阶稳定解时取得了最优的O(ε⁻³)样本复杂度。然而，关于拟牛顿方法的研究仍存在空白。鉴于拟牛顿方法具有更高精度和更强鲁棒性，本文针对光滑性存在非均匀性的情况，提出了一种快速随机拟牛顿方法。通过结合梯度裁剪与方差缩减技术，我们的算法达到了目前已知最优的O(ε⁻³)样本复杂度，并通过简单的超参数调优实现了收敛加速。数值实验表明，本文提出的算法优于现有的先进方法。