Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge. To tackle this challenge, in this paper, we propose a novel parameter-free optimizer, \textsc{AdamG} (Adam with the golden step size), designed to automatically adapt to diverse optimization problems without manual tuning. The core technique underlying \textsc{AdamG} is our golden step size derived for the AdaGrad-Norm algorithm, which is expected to help AdaGrad-Norm preserve the tuning-free convergence and approximate the optimal step size in expectation w.r.t. various optimization scenarios. To better evaluate tuning-free performance, we propose a novel evaluation criterion, \textit{reliability}, to comprehensively assess the efficacy of parameter-free optimizers in addition to classical performance criteria. Empirical results demonstrate that compared with other parameter-free baselines, \textsc{AdamG} achieves superior performance, which is consistently on par with Adam using a manually tuned learning rate across various optimization tasks.

翻译：超参数调优，尤其是在自适应梯度训练方法中选择合适的学习率，仍然是一个挑战。为应对这一挑战，本文提出了一种新颖的无参数优化器 \textsc{AdamG}（采用黄金步长的Adam），旨在无需手动调优即可自动适应不同的优化问题。\textsc{AdamG} 的核心技术是我们为 AdaGrad-Norm 算法推导出的黄金步长，该步长有望帮助 AdaGrad-Norm 保持免调优的收敛性，并在期望意义上逼近各种优化场景下的最优步长。为了更好地评估免调优性能，我们提出了一种新的评估标准——\textit{可靠性}，以在经典性能标准之外，全面评估无参数优化器的效能。实证结果表明，与其他无参数基线方法相比，\textsc{AdamG} 取得了更优的性能，在各种优化任务中，其表现始终与使用手动调优学习率的 Adam 相当。