We study the discrete dynamics of mini-batch gradient descent for least squares regression when sampling without replacement. We show that the dynamics and generalization error of mini-batch gradient descent depends on a sample cross-covariance matrix $Z$ between the original features $X$ and a set of new features $\widetilde{X}$, in which each feature is modified by the mini-batches that appear before it during the learning process in an averaged way. Using this representation, we rigorously establish that the dynamics of mini-batch and full-batch gradient descent agree up to leading order with respect to the step size using the linear scaling rule. We also study discretization effects that a continuous-time gradient flow analysis cannot detect, and show that mini-batch gradient descent converges to a step-size dependent solution, in contrast with full-batch gradient descent. Finally, we investigate the effects of batching, assuming a random matrix model, by using tools from free probability theory to numerically compute the spectrum of $Z$.

翻译：我们研究了无放回采样条件下，最小二乘回归问题中小批量梯度下降的离散动力学行为。研究表明，小批量梯度下降的动力学特性与泛化误差取决于原始特征矩阵$X$与一组新特征矩阵$\widetilde{X}$之间的样本互协方差矩阵$Z$，其中每个特征在学习过程中会以平均化的方式受到其之前出现的小批量的修正。基于该表征，我们严格证明了在使用线性缩放规则时，小批量与全批量梯度下降的动力学行为在步长的一阶近似下保持一致。同时，我们研究了连续时间梯度流分析无法捕捉的离散化效应，发现小批量梯度下降会收敛至依赖步长的解，这与全批量梯度下降的行为形成对比。最后，在随机矩阵模型的假设下，我们借助自由概率论工具数值计算了矩阵$Z$的谱分布，以此探究批处理机制的影响。