We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width $k$ and input dimension $d$ are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size $n$ scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters $kd+k$ and of data $n$ are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate $n/d^2$, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms.

翻译：我们考虑一个师生模型的监督学习框架，其中使用了一个完全训练的双层神经网络，其宽度$k$和输入维度$d$均较大且成比例。我们提出了一种有效理论，用于近似网络在任意激活函数下的贝叶斯最优泛化误差，该理论适用于样本量$n$与输入维度呈二次方比例增长的机制，即围绕插值阈值附近，此时可训练参数数量$kd+k$与数据量$n$相当。我们的分析处理了通用的权重分布。我们发现了一个不连续的相变，将“通用”相与“专化”相分离开来。在通用相中，泛化误差与权重分布无关，并随着采样率$n/d^2$缓慢衰减，学生仅学习教师权重的一些非线性组合。在专化相中，误差依赖于权重分布，并且由于学生网络向教师网络对齐而衰减得更快。因此，我们揭示了在插值附近存在一个高度可预测的解，然而该解可能难以通过实际算法找到。