We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.

翻译：我们提出了一种在小学习率条件下随机梯度下降的新极限动力学，称为随机修正流。这类由柱形布朗运动驱动的随机微分方程，通过引入规则扩散系数并匹配多点统计特性，改进了原有的随机修正方程。作为第二项贡献，我们引入了分布依赖型随机修正流，并证明其描述了在小学习率-无限宽度缩放机制下随机梯度下降的波动极限动力学。