In recent studies, line search methods have shown significant improvements in the performance of traditional stochastic gradient descent techniques, eliminating the need for a specific learning rate schedule. In this paper, we identify existing issues in state-of-the-art line search methods, propose enhancements, and rigorously evaluate their effectiveness. We test these methods on larger datasets and more complex data domains than before. Specifically, we improve the Armijo line search by integrating the momentum term from ADAM in its search direction, enabling efficient large-scale training, a task that was previously prone to failure using Armijo line search methods. Our optimization approach outperforms both the previous Armijo implementation and tuned learning rate schedules for Adam. Our evaluation focuses on Transformers and CNNs in the domains of NLP and image data. Our work is publicly available as a Python package, which provides a hyperparameter free Pytorch optimizer.

翻译：近期研究表明，线搜索方法显著提升了传统随机梯度下降技术的性能，消除了对特定学习率调度方案的需求。本文识别了当前最先进线搜索方法中存在的若干问题，提出了改进方案，并对其有效性进行了严格评估。我们在更大规模数据集及更复杂的数据域中测试了这些方法。具体而言，我们通过将ADAM中的动量项融入搜索方向，改进了Armijo线搜索，从而实现了此前使用Armijo线搜索方法容易失败的大规模高效训练。我们的优化方法在性能上超越了先前的Armijo实现以及为Adam调优的学习率调度方案。评估重点聚焦于自然语言处理与图像数据领域的Transformer和CNN模型。本研究以Python包形式公开发布，提供了一个无需超参数的PyTorch优化器。