Adam [Kingma & Ba, 2015] is the de facto optimizer in deep learning, yet its theoretical understanding remains limited. Prior analyses show that Adam favors solutions aligned with $\ell_\infty$-geometry, but these results are restricted to the full-batch regime. In this work, we study the implicit bias of incremental Adam (using one sample per step) for logistic regression on linearly separable data, and show that its bias can deviate from the full-batch behavior. As an extreme example, we construct datasets on which incremental Adam provably converges to the $\ell_2$-max-margin classifier, in contrast to the $\ell_\infty$-max-margin bias of full-batch Adam. For general datasets, we characterize its bias using a proxy algorithm for the $β_2 \to 1$ limit. This proxy maximizes a data-adaptive Mahalanobis-norm margin, whose associated covariance matrix is determined by a data-dependent dual fixed-point formulation. We further present concrete datasets where this bias reduces to the standard $\ell_2$- and $\ell_\infty$-max-margin classifiers. As a counterpoint, we prove that Signum [Bernstein et al., 2018] converges to the $\ell_\infty$-max-margin classifier for any batch size. Overall, our results highlight that the implicit bias of Adam crucially depends on both the batching scheme and the dataset, while Signum remains invariant.

翻译：Adam [Kingma & Ba, 2015] 是深度学习中的事实标准优化器，但其理论理解仍然有限。先前分析表明Adam倾向于与$\ell_\infty$几何对齐的解，但这些结果仅限于全批次机制。在本工作中，我们研究了在线性可分数据上用于逻辑回归的增量Adam（每步使用一个样本）的隐式偏差，并证明其偏差可能偏离全批次行为。作为一个极端示例，我们构建了数据集，证明增量Adam可收敛到$\ell_2$-最大间隔分类器，这与全批次Adam的$\ell_\infty$-最大间隔偏差形成对比。对于一般数据集，我们通过$β_2 \to 1$极限的代理算法来刻画其偏差。该代理算法最大化数据自适应的马氏范数间隔，其关联协方差矩阵由数据依赖的对偶定点公式确定。我们进一步展示了具体数据集，其中该偏差可简化为标准$\ell_2$-和$\ell_\infty$-最大间隔分类器。作为对照，我们证明Signum [Bernstein et al., 2018] 对于任何批次大小都收敛到$\ell_\infty$-最大间隔分类器。总体而言，我们的结果强调Adam的隐式偏差关键取决于批处理方案和数据集，而Signum则保持不变。