In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization. To reveal this bias, we identify invariant sets, or subsets of parameter space that remain unmodified by SGD. We focus on two classes of invariant sets that correspond to simpler (sparse or low-rank) subnetworks and commonly appear in modern architectures. Our analysis uncovers that SGD exhibits a property of stochastic attractivity towards these simpler invariant sets. We establish a sufficient condition for stochastic attractivity based on a competition between the loss landscape's curvature around the invariant set and the noise introduced by stochastic gradients. Remarkably, we find that an increased level of noise strengthens attractivity, leading to the emergence of attractive invariant sets associated with saddle-points or local maxima of the train loss. We observe empirically the existence of attractive invariant sets in trained deep neural networks, implying that SGD dynamics often collapses to simple subnetworks with either vanishing or redundant neurons. We further demonstrate how this simplifying process of stochastic collapse benefits generalization in a linear teacher-student framework. Finally, through this analysis, we mechanistically explain why early training with large learning rates for extended periods benefits subsequent generalization.

翻译：本文揭示了随机梯度下降（SGD）的一种强隐式偏差，该偏差促使过度表达的网络向更简单的子网络大幅简化，从而显著减少独立参数数量并提升泛化性能。为揭示这一偏差，我们识别了参数空间中不受SGD更新的不变集合，重点关注两类对应于更简单（稀疏或低秩）子网络且常见于现代架构的不变集合。分析表明，SGD对这些简单不变集合具备随机吸引性质。我们基于损失函数曲面在不变集合周围的曲率与随机梯度引入的噪声之间的竞争关系，建立了随机吸引的充分条件。值得注意的是，我们发现噪声强度的增加会增强吸引性，导致与训练损失鞍点或局部极大值相关的吸引不变集合涌现。通过实验观察，训练后的深度神经网络中确实存在吸引不变集合，这意味着SGD动力学常坍缩至神经元消失或冗余的简单子网络。我们进一步在线性师生框架下证明了这种随机坍缩过程对泛化的促进作用。最终，基于此分析，我们从机理上解释了为何早期使用大学习率进行长时间训练有利于后续泛化性能提升。