Optimizing the learning rate remains a critical challenge in machine learning, essential for achieving model stability and efficient convergence. The Vector Auxiliary Variable (VAV) algorithm introduces a novel energy-based self-adjustable learning rate optimization method designed for unconstrained optimization problems. It incorporates an auxiliary variable $r$ to facilitate efficient energy approximation without backtracking while adhering to the unconditional energy dissipation law. Notably, VAV demonstrates superior stability with larger learning rates and achieves faster convergence in the early stage of the training process. Comparative analyses demonstrate that VAV outperforms Stochastic Gradient Descent (SGD) across various tasks. This paper also provides rigorous proof of the energy dissipation law and establishes the convergence of the algorithm under reasonable assumptions. Additionally, $r$ acts as an empirical lower bound of the training loss in practice, offering a novel scheduling approach that further enhances algorithm performance.

翻译：优化学习率仍然是机器学习中的关键挑战，对于实现模型稳定性和高效收敛至关重要。向量辅助变量（VAV）算法提出了一种新颖的基于能量的自调节学习率优化方法，专为无约束优化问题设计。该方法引入辅助变量$r$，在不回溯的情况下实现高效的能量近似，同时遵循无条件能量耗散定律。值得注意的是，VAV在较大学习率下表现出卓越的稳定性，并在训练过程早期实现更快收敛。对比分析表明，VAV在各种任务中均优于随机梯度下降（SGD）。本文还严格证明了能量耗散定律，并在合理假设下建立了算法的收敛性。此外，实践中$r$可作为训练损失的经验下界，提供了一种新颖的调度方法，进一步提升了算法性能。