Stochastic gradient descent plays a fundamental role in nearly all applications of deep learning. However its ability to converge to a global minimum remains shrouded in mystery. In this paper we propose to study the behavior of the loss function on fixed mini-batches along SGD trajectories. We show that the loss function on a fixed batch appears to be remarkably convex-like. In particular for ResNet the loss for any fixed mini-batch can be accurately modeled by a quadratic function and a very low loss value can be reached in just one step of gradient descent with sufficiently large learning rate. We propose a simple model that allows to analyze the relationship between the gradients of stochastic mini-batches and the full batch. Our analysis allows us to discover the equivalency between iterate aggregates and specific learning rate schedules. In particular, for Exponential Moving Average (EMA) and Stochastic Weight Averaging we show that our proposed model matches the observed training trajectories on ImageNet. Our theoretical model predicts that an even simpler averaging technique, averaging just two points a many steps apart, significantly improves accuracy compared to the baseline. We validated our findings on ImageNet and other datasets using ResNet architecture.

翻译：随机梯度下降在几乎所有深度学习应用中扮演着基础性角色。然而，其收敛到全局最小值的能力仍笼罩在神秘之中。本文提出研究沿随机梯度下降轨迹的固定小批量上损失函数的行为。我们表明，固定小批量上的损失函数呈现出显著类凸性。特别地，对于ResNet，任何固定小批量的损失都可以通过二次函数精确建模，且仅需一步足够大学习率的梯度下降即可达到极低损失值。我们提出一个简单模型，用于分析随机小批量梯度与全批量梯度之间的关系。该分析使我们能够发现迭代聚合与特定学习率调度之间的等价性。特别地，对于指数移动平均（EMA）和随机权重平均，我们展示所提模型与ImageNet上观测到的训练轨迹吻合。我们的理论模型预测，一种更简单的平均技术——仅平均相隔多个步长的两个点——相比基线显著提升准确率。我们使用ResNet架构在ImageNet及其他数据集上验证了这一发现。