Adam has become one of the most favored optimizers in deep learning problems. Despite its success in practice, numerous mysteries persist regarding its theoretical understanding. In this paper, we study the implicit bias of Adam in linear logistic regression. Specifically, we show that when the training data are linearly separable, Adam converges towards a linear classifier that achieves the maximum $\ell_\infty$-margin. Notably, for a general class of diminishing learning rates, this convergence occurs within polynomial time. Our result shed light on the difference between Adam and (stochastic) gradient descent from a theoretical perspective.

翻译：Adam已成为深度学习问题中最受青睐的优化器之一。尽管在实践中取得了成功，但其理论理解仍存在诸多未解之谜。本文研究了Adam在线性逻辑回归中的隐式偏差。具体而言，我们证明了当训练数据线性可分时，Adam会收敛于一个实现最大$\ell_\infty$-间隔的线性分类器。值得注意的是，对于一大类递减学习率，该收敛过程在多项式时间内完成。我们的结果从理论视角揭示了Adam与（随机）梯度下降之间的差异。