Many recent works have studied the eigenvalue spectrum of the Conjugate Kernel (CK) defined by the nonlinear feature map of a feedforward neural network. However, existing results only establish weak convergence of the empirical eigenvalue distribution, and fall short of providing precise quantitative characterizations of the ''spike'' eigenvalues and eigenvectors that often capture the low-dimensional signal structure of the learning problem. In this work, we characterize these signal eigenvalues and eigenvectors for a nonlinear version of the spiked covariance model, including the CK as a special case. Using this general result, we give a quantitative description of how spiked eigenstructure in the input data propagates through the hidden layers of a neural network with random weights. As a second application, we study a simple regime of representation learning where the weight matrix develops a rank-one signal component over training and characterize the alignment of the target function with the spike eigenvector of the CK on test data.

翻译：许多近期工作研究了前馈神经网络非线性特征映射定义的共轭核（CK）的经验特征值谱。然而，现有结果仅建立了经验特征值分布的弱收敛性，未能对通常捕捉学习问题低维信号结构的"尖峰"特征值和特征向量提供精确的定量刻画。本研究针对非线性版本的尖峰协方差模型（包括CK作为特例）刻画了这些信号特征值和特征向量。基于这一通用结果，我们定量描述了输入数据中的尖峰特征结构如何通过随机权重神经网络隐藏层传播。作为第二项应用，我们研究了表征学习的简单机制——其中权重矩阵在训练过程中形成秩一信号分量——并刻画了测试数据上目标函数与CK尖峰特征向量的对齐程度。