The Exponential Moving Average (EMA) is a cornerstone of widely used optimizers such as Adam. However, existing theoretical analyses of Adam-style methods have notable limitations: their guarantees can remain suboptimal in the zero-noise regime, rely on restrictive boundedness conditions (e.g., bounded gradients or objective gaps), use constant or open-loop stepsizes, or require prior knowledge of Lipschitz constants. To overcome these bottlenecks, we introduce OptEMA and analyze two novel variants: OptEMA-M, which applies an adaptive, decreasing EMA coefficient to the first-order moment with a fixed second-order decay, and OptEMA-V, which swaps these roles. Crucially, OptEMA is closed-loop and Lipschitz-free in the sense that its effective stepsizes are trajectory-dependent and do not require the Lipschitz constant for parameterization. Under standard stochastic gradient descent (SGD) assumptions, namely smoothness, a lower-bounded objective, and unbiased gradients with bounded variance, we establish rigorous convergence guarantees. Both variants achieve a noise-adaptive convergence rate of $\widetilde{\mathcal{O}}(T^{-1/2}+σ^{1/2} T^{-1/4})$ for the average gradient norm, where $σ$ is the noise level. In particular, in the zero-noise regime where $σ=0$, our bounds reduce to the nearly optimal deterministic rate $\widetilde{\mathcal{O}}(T^{-1/2})$ without manual hyperparameter retuning.

翻译：指数移动平均（EMA）是Adam等广泛使用的优化器的基石。然而，现有对Adam类方法的理论分析存在显著局限：其保证在零噪声情形下可能仍非最优，依赖于限制性的有界条件（例如有界梯度或目标函数间隙），使用恒定或开环步长，或需要李普希茨常数的先验知识。为克服这些瓶颈，我们提出了OptEMA并分析了两种新颖变体：OptEMA-M，其对一阶矩应用自适应递减的EMA系数并固定二阶衰减；以及OptEMA-V，其交换了二者的角色。关键在于，OptEMA是闭环且无李普希茨依赖的，其有效步长依赖于优化轨迹且无需李普希茨常数进行参数化。在标准随机梯度下降（SGD）假设下，即光滑性、目标函数有下界、梯度无偏且方差有界，我们建立了严格的收敛性保证。两种变体均实现了平均梯度范数的噪声自适应收敛速率$\widetilde{\mathcal{O}}(T^{-1/2}+σ^{1/2} T^{-1/4})$，其中$σ$为噪声水平。特别地，在零噪声情形（$σ=0$）下，我们的界可简化为近乎最优的确定性速率$\widetilde{\mathcal{O}}(T^{-1/2})$，而无需手动重新调整超参数。