In this paper, we investigate a two-layer fully connected neural network of the form $f(X)=\frac{1}{\sqrt{d_1}}\boldsymbol{a}^\top \sigma\left(WX\right)$, where $X\in\mathbb{R}^{d_0\times n}$ is a deterministic data matrix, $W\in\mathbb{R}^{d_1\times d_0}$ and $\boldsymbol{a}\in\mathbb{R}^{d_1}$ are random Gaussian weights, and $\sigma$ is a nonlinear activation function. We study the limiting spectral distributions of two empirical kernel matrices associated with $f(X)$: the empirical conjugate kernel (CK) and neural tangent kernel (NTK), beyond the linear-width regime ($d_1\asymp n$). We focus on the $\textit{ultra-wide regime}$, where the width $d_1$ of the first layer is much larger than the sample size $n$. Under appropriate assumptions on $X$ and $\sigma$, a deformed semicircle law emerges as $d_1/n\to\infty$ and $n\to\infty$. We first prove this limiting law for generalized sample covariance matrices with some dependency. To specify it for our neural network model, we provide a nonlinear Hanson-Wright inequality that is suitable for neural networks with random weights and Lipschitz activation functions. We also demonstrate non-asymptotic concentrations of the empirical CK and NTK around their limiting kernels in the spectral norm, along with lower bounds on their smallest eigenvalues. As an application, we show that random feature regression induced by the empirical kernel achieves the same asymptotic performance as its limiting kernel regression under the ultra-wide regime. This allows us to calculate the asymptotic training and test errors for random feature regression using the corresponding kernel regression.

翻译：本文研究形如 $f(X)=\frac{1}{\sqrt{d_1}}\boldsymbol{a}^\top \sigma\left(WX\right)$ 的两层全连接神经网络，其中 $X\in\mathbb{R}^{d_0\times n}$ 为确定性数据矩阵，$W\in\mathbb{R}^{d_1\times d_0}$ 和 $\boldsymbol{a}\in\mathbb{R}^{d_1}$ 为随机高斯权重，$\sigma$ 为非线性激活函数。我们在线性宽度领域（$d_1\asymp n$）之外，研究 $f(X)$ 关联的两个经验核矩阵——经验共轭核（CK）和神经正切核（NTK）的极限谱分布。重点关注宽层 $d_1$ 远大于样本量 $n$ 的超宽领域。在 $X$ 和 $\sigma$ 的适当假设下，当 $d_1/n\to\infty$ 且 $n\to\infty$ 时，会出现变形半圆律。我们首先证明该极限律适用于具有某种依赖性的广义样本协方差矩阵。为将其具体化到我们的神经网络模型中，我们提出了适用于随机权重和 Lipschitz 激活函数的非线性 Hanson-Wright 不等式。我们还证明了经验 CK 和 NTK 在谱范数下围绕其极限核的非渐近集中性，以及其最小特征值的下界。作为应用，我们证明在超宽领域中，由经验核诱导的随机特征回归能达到与其极限核回归相同的渐近性能。这使得我们能够利用相应的核回归计算随机特征回归的渐近训练误差和测试误差。