In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization. To reveal this bias, we identify invariant sets, or subsets of parameter space that remain unmodified by SGD. We focus on two classes of invariant sets that correspond to simpler (sparse or low-rank) subnetworks and commonly appear in modern architectures. Our analysis uncovers that SGD exhibits a property of stochastic attractivity towards these simpler invariant sets. We establish a sufficient condition for stochastic attractivity based on a competition between the loss landscape's curvature around the invariant set and the noise introduced by stochastic gradients. Remarkably, we find that an increased level of noise strengthens attractivity, leading to the emergence of attractive invariant sets associated with saddle-points or local maxima of the train loss. We observe empirically the existence of attractive invariant sets in trained deep neural networks, implying that SGD dynamics often collapses to simple subnetworks with either vanishing or redundant neurons. We further demonstrate how this simplifying process of stochastic collapse benefits generalization in a linear teacher-student framework. Finally, through this analysis, we mechanistically explain why early training with large learning rates for extended periods benefits subsequent generalization.

翻译：在本研究中，我们揭示了随机梯度下降（SGD）的一种强烈隐式偏好，这种偏好会驱使过度复杂的网络向更简单的子网络演化，从而显著减少独立参数的数量并提升泛化能力。为揭示这种偏好，我们识别了不变集——即参数空间中那些在SGD迭代过程中保持不变的子集。我们重点关注两类对应于更简单（稀疏或低秩）子网络的不变集，这两类结构在现代神经网络架构中普遍存在。我们的分析发现，SGD表现出向这些简单不变集随机吸引的特性。我们基于损失函数在不变集附近的曲率与随机梯度引入的噪声之间的竞争关系，建立了随机吸引性的充分条件。值得注意的是，我们发现更高水平的噪声会增强吸引性，导致与训练损失函数的鞍点或局部极大值相关联的吸引性不变集的出现。我们通过实验观测到，在训练后的深度神经网络中存在吸引性不变集，这意味着SGD动态常常会坍缩到具有消失神经元或冗余神经元的简单子网络。我们进一步在线性师生框架中证明了这种随机坍缩的简化过程如何有益于泛化性能。最后，通过该分析，我们从机制上解释了为何在早期训练阶段长时间使用较大学习率有利于后续的泛化表现。