Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of second-order, curvature-based methods (such as quasi-Newton methods and K-FAC). Noting that second-order methods often only function effectively with the addition of stabilising heuristics (such as Levenberg-Marquardt damping), we ask how much these (as opposed to the second-order curvature model) contribute to second-order algorithms' performance. We thus study AdamQLR: an optimiser combining damping and learning rate selection techniques from K-FAC (Martens & Grosse, 2015) with the update directions proposed by Adam, inspired by considering Adam through a second-order lens. We evaluate AdamQLR on a range of regression and classification tasks at various scales and hyperparameter tuning methodologies, concluding K-FAC's adaptive heuristics are of variable standalone general effectiveness, and finding an untuned AdamQLR setting can achieve comparable performance vs runtime to tuned benchmarks.

翻译：深度学习优化研究的一个核心矛盾在于：基于一阶梯度的方法（如SGD和Adam）具有计算效率优势，而基于二阶曲率的方法（如拟牛顿法与K-FAC）则具备理论效率优势。注意到二阶方法通常需要引入稳定化启发式策略（如Levenberg-Marquardt阻尼）才能有效运行，我们试图探究这些策略（相对于二阶曲率模型本身）对二阶算法性能的实际贡献。为此，我们提出AdamQLR优化器：该算法融合了K-FAC（Martens & Grosse, 2015）中的阻尼与学习率选择技术，同时采用Adam提出的更新方向，其设计灵感源于通过二阶视角重新审视Adam。我们在不同规模的回归与分类任务上，采用多种超参数调优方法对AdamQLR进行评估。结论表明：K-FAC的自适应启发式策略在不同场景下的独立泛化效果存在差异，同时发现未经调优的AdamQLR配置在性能与运行时间的权衡上能够达到与调优基准方法相当的水平。