Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a hundred different optimizers, yet there is scarce data on optimizer performance in various tasks. In particular, none of the benchmarks test the performance of optimizers on source code-related problems. However, existing benchmark data indicates that certain optimizers may be more efficient for particular domains. In this work, we test the performance of various optimizers on deep learning models for source code and find that the choice of an optimizer can have a significant impact on the model quality, with up to two-fold score differences between some of the relatively well-performing optimizers. We also find that RAdam optimizer (and its modification with the Lookahead envelope) is the best optimizer that almost always performs well on the tasks we consider. Our findings show a need for a more extensive study of the optimizers in code-related tasks, and indicate that the ML4SE community should consider using RAdam instead of Adam as the default optimizer for code-related deep learning tasks.

翻译：使用深度学习模型解决问题需要研究人员采用特定的优化方法来优化损失函数。研究社区已开发出上百种不同的优化器，但关于优化器在各种任务中性能的数据仍然匮乏。特别是，当前尚无基准测试评估优化器在源代码相关问题上的表现。然而，现有基准数据表明，某些优化器可能在特定领域更为高效。在本研究中，我们测试了多种优化器在面向源代码的深度学习模型上的性能，发现优化器的选择对模型质量有显著影响，部分表现相对优异的优化器之间分数差异可达两倍。我们还发现，RAdam优化器（及其结合Lookahead机制的改进版本）几乎总是能在我们考虑的任务中表现优异，是最优选择。我们的研究结果表明，需要更广泛地研究代码相关任务中的优化器，并提示ML4SE社区应考虑使用RAdam替代Adam作为代码相关深度学习任务的默认优化器。