For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where the input dimension, hidden layer widths, and number of training samples are proportionally large. This characterization is formulated in terms of the population covariance of the features. Our work is partially motivated by the problem of learning with Gaussian rainbow neural networks, namely deep non-linear fully-connected networks with random but structured weights, whose row-wise covariances are further allowed to depend on the weights of previous layers. For such networks we also derive a closed-form formula for the feature covariance in terms of the weight matrices. We further find that in some cases our results can capture feature maps learned by deep, finite-width neural networks trained under gradient descent.

翻译：对于一大类特征映射，我们提供了测试误差的紧致渐近刻画，该误差与学习读出层相关，在高维极限下（输入维度、隐藏层宽度及训练样本数量成比例增大）。该刻画以特征的总体协方差为基础。本研究部分动机来源于高斯彩虹神经网络的学习问题，即具有随机但结构化权重的深度非线性全连接网络，其行协方差进一步允许依赖于前一层权重。针对此类网络，我们还推导了基于权重矩阵的特征协方差闭合表达式。进一步发现，在某些情况下我们的结果能够捕捉经梯度下降训练后的深度有限宽度神经网络所学习到的特征映射。