We present new algorithms for optimizing non-smooth, non-convex stochastic objectives based on a novel analysis technique. This improves the current best-known complexity for finding a $(\delta,\epsilon)$-stationary point from $O(\epsilon^{-4}\delta^{-1})$ stochastic gradient queries to $O(\epsilon^{-3}\delta^{-1})$, which we also show to be optimal. Our primary technique is a reduction from non-smooth non-convex optimization to online learning, after which our results follow from standard regret bounds in online learning. For deterministic and second-order smooth objectives, applying more advanced optimistic online learning techniques enables a new complexity of $O(\epsilon^{-1.5}\delta^{-0.5})$. Our techniques also recover all optimal or best-known results for finding $\epsilon$ stationary points of smooth or second-order smooth objectives in both stochastic and deterministic settings.

翻译：我们提出了一种基于新型分析技术的算法，用于优化非光滑、非凸的随机目标函数。该算法将当前寻找$(\delta,\epsilon)$-驻点的最优随机梯度查询复杂度从$O(\epsilon^{-4}\delta^{-1})$改进至$O(\epsilon^{-3}\delta^{-1})$，并证明该复杂度是最优的。核心技术是将非光滑非凸优化问题归约到在线学习框架，随后通过在线学习中的标准遗憾界获得结论。对于确定性和二阶光滑目标函数，应用更先进的乐观在线学习技术可将复杂度降至$O(\epsilon^{-1.5}\delta^{-0.5})$。该技术框架还恢复了在随机和确定性设定下，光滑或二阶光滑目标函数寻找$\epsilon$-驻点的所有最优或已知最佳结果。