We introduce a new class of adaptive non-linear autoregressive (Nlar) models incorporating the concept of momentum, which dynamically estimate both the learning rates and momentum as the number of iterations increases. In our method, the growth of the gradients is controlled using a scaling (clipping) function, leading to stable convergence. Within this framework, we propose three distinct estimators for learning rates and provide theoretical proof of their convergence. We further demonstrate how these estimators underpin the development of effective Nlar optimizers. The performance of the proposed estimators and optimizers is rigorously evaluated through extensive experiments across several datasets and a reinforcement learning environment. The results highlight two key features of the Nlar optimizers: robust convergence despite variations in underlying parameters, including large initial learning rates, and strong adaptability with rapid convergence during the initial epochs.

翻译：我们引入了一类融合动量概念的新型自适应非线性自回归模型，该模型能够随着迭代次数的增加动态估计学习率与动量。在本方法中，梯度的增长通过缩放函数进行控制，从而确保收敛过程的稳定性。在此框架下，我们提出了三种不同的学习率估计器，并提供了其收敛性的理论证明。我们进一步阐述了这些估计器如何支撑高效非线性自回归优化器的开发。通过在多个数据集及强化学习环境中进行大量实验，对所提出的估计器与优化器的性能进行了严格评估。结果凸显了非线性自回归优化器的两大关键特性：即使在基础参数存在差异的情况下仍能保持稳健收敛，以及在前几个训练周期内展现出强大的适应性与快速收敛能力。