In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model the trained network by a spiked Random Features (sRF) model. Further building on recent progress on Gaussian universality (Dandi et al., 2023), we provide an exact asymptotic description of the generalization error of the sRF in the high-dimensional limit where the number of samples, the width, and the input dimension grow at a proportional rate. The resulting characterization for sRFs also captures closely the learning curves of the original network model. This enables us to understand how adapting to the data is crucial for the network to efficiently learn non-linear functions in the direction of the gradient -- where at initialization it can only express linear functions in this regime.

翻译：本文研究了两层神经网络在单次梯度下降训练后如何从数据中学习特征，并超越核机制的性能。借鉴（Ba等人，2022）的见解，我们通过尖峰随机特征模型对训练后的网络进行建模。进一步基于高斯普适性（Dandi等人，2023）的最新进展，我们在样本数、网络宽度与输入维度成比例增长的高维极限下，给出了sRF模型泛化误差的精确渐近描述。该sRF模型的表征结果也能准确捕捉原始网络模型的学习曲线。这使我们得以理解：网络为高效学习梯度方向上的非线性函数（在此机制下初始化时仅能表达该方向的线性函数），其适应数据的能力至关重要。