We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance. In particular, we demonstrate a deterministic equivalent of SGD in the form of a system of ordinary differential equations that describes a wide class of statistics, such as the risk and other measures of sub-optimality. This equivalence holds with overwhelming probability when the model parameter count grows proportionally to the number of data. This framework allows us to obtain learning rate thresholds for stability of SGD as well as convergence guarantees. In addition to the deterministic equivalent, we introduce an SDE with a simplified diffusion coefficient (homogenized SGD) which allows us to analyze the dynamics of general statistics of SGD iterates. Finally, we illustrate this theory on some standard examples and show numerical simulations which give an excellent match to the theory.

翻译：我们分析了流式随机梯度下降（SGD）在高维极限下的动力学行为，该行为适用于广义线性模型和多指标模型（例如逻辑回归、相位恢复），并考虑了一般数据协方差结构。特别地，我们证明了SGD的一个确定等价形式，即一组常微分方程系统，该系统描述了广泛统计量（如风险及其他次优性度量）的行为。当模型参数数量与数据数量成比例增长时，这一等价性以压倒性概率成立。该框架使我们能够获得SGD稳定性的学习率阈值以及收敛性保证。除确定等价形式外，我们还引入了一个具有简化扩散系数的随机微分方程（均质化SGD），从而能够分析SGD迭代过程中一般统计量的动力学。最后，我们通过若干标准示例对理论进行了说明，并展示了与理论高度吻合的数值模拟结果。