Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of second-order, curvature-based methods (such as quasi-Newton methods and K-FAC). Noting that second-order methods often only function effectively with the addition of stabilising heuristics (such as Levenberg-Marquardt damping), we ask how much these (as opposed to the second-order curvature model) contribute to second-order algorithms' performance. We thus study AdamQLR: an optimiser combining damping and learning rate selection techniques from K-FAC (Martens & Grosse, 2015) with the update directions proposed by Adam, inspired by considering Adam through a second-order lens. We evaluate AdamQLR on a range of regression and classification tasks at various scales and hyperparameter tuning methodologies, concluding K-FAC's adaptive heuristics are of variable standalone general effectiveness, and finding an untuned AdamQLR setting can achieve comparable performance vs runtime to tuned benchmarks.

翻译：深度学习优化研究的特点在于一阶梯度方法（如SGD和Adam）的计算效率与二阶曲率方法（如拟牛顿法和K-FAC）的理论效率之间的张力。注意到二阶方法通常仅在加入稳定化启发式策略（如Levenberg-Marquardt阻尼）后才能有效运行，我们探究这些策略（相对于二阶曲率模型本身）对二阶算法性能的贡献程度。为此，我们研究了AdamQLR：该优化器融合了K-FAC（Martens & Grosse, 2015）的阻尼与学习率选择技术，并结合了Adam提出的更新方向，其设计灵感源于通过二阶视角审视Adam。我们在不同规模与超参数调优方法下，通过一系列回归和分类任务评估AdamQLR，最终得出K-FAC的自适应启发式策略具有差异化的独立泛化效能，并发现未经调优的AdamQLR配置能够在性能与运行时间方面达到与调优基准相当的水平。