We investigate the training dynamics of two-layer neural networks when learning multi-index target functions. We focus on multi-pass gradient descent (GD) that reuses the batches multiple times and show that it significantly changes the conclusion about which functions are learnable compared to single-pass gradient descent. In particular, multi-pass GD with finite stepsize is found to overcome the limitations of gradient flow and single-pass GD given by the information exponent (Ben Arous et al., 2021) and leap exponent (Abbe et al., 2023) of the target function. We show that upon re-using batches, the network achieves in just two time steps an overlap with the target subspace even for functions not satisfying the staircase property (Abbe et al., 2021). We characterize the (broad) class of functions efficiently learned in finite time. The proof of our results is based on the analysis of the Dynamical Mean-Field Theory (DMFT). We further provide a closed-form description of the dynamical process of the low-dimensional projections of the weights, and numerical experiments illustrating the theory.

翻译：我们研究了双层神经网络在学习多指标目标函数时的训练动力学。关注多轮梯度下降（GD）——即重用批次多次，并表明其与单轮梯度下降相比，显著改变了对哪些函数可学习的结论。特别地，我们发现使用有限步长的多轮GD能够克服梯度流和单轮GD的局限性，这些局限性由目标函数的信息指数（Ben Arous等人，2021）和跳跃指数（Abbe等人，2023）决定。我们证明，即使对于不满足阶梯性质（Abbe等人，2021）的函数，通过重用批次，网络仅在两个时间步内就能实现与目标子空间的重叠。我们刻画了在有限时间内高效学习的（宽泛）函数类。证明基于动力学平均场理论（DMFT）的分析。我们进一步给出了权重的低维投影动力学过程的封闭形式描述，以及用于说明该理论的数值实验。