We study the problem of learning a low-degree spherical polynomial of degree $\ell_0 = Θ(1) \ge 1$ defined on the unit sphere in $\RR^d$ by training an over-parameterized two-layer neural network (NN) with channel attention in this paper. Our main result is the significantly improved sample complexity for learning such low-degree polynomials. We show that, for any regression risk $\eps \in (0,1)$, a carefully designed two-layer NN with channel attention and finite width of $m \ge Θ({n^4 \log (2n/δ)}/{d^{2\ell_0}})$ trained by the vanilla gradient descent (GD) requires the lowest sample complexity of $n \asymp Θ(d^{\ell_0}/\eps)$ with probability $1-δ$ for every $δ\in (0,1)$, in contrast with the representative sample complexity $Θ\pth{d^{\ell_0} \max\set{\eps^{-2},\log d}}$, where $n$ is the training daata size. Moreover, such sample complexity is not improvable since the trained network renders a sharp rate of the nonparametric regression risk of the order $Θ(d^{\ell_0}/{n})$ with probability at least $1-δ$. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank $Θ(d^{\ell_0})$ is $Θ(d^{\ell_0}/{n})$, so that the rate of the nonparametric regression risk of the network trained by GD is minimax optimal. The training of the two-layer NN with channel attention consists of two stages. In Stage 1, a provable learnable channel selection algorithm identifies the ground-truth channel number $\ell_0$ from the initial $L \ge \ell_0$ channels in the first-layer activation, with high probability. This learnable selection is achieved by an efficient one-step GD update on both layers, enabling feature learning for low-degree polynomial targets. In Stage 2, the second layer is trained by standard GD using the activation function with the selected channels.

翻译：本文研究通过训练具有通道注意力的过参数化双层神经网络来学习定义在$\RR^d$单位球面上的低阶球面多项式（阶数$\ell_0 = Θ(1) \ge 1$）的问题。我们的主要结果是显著改善了学习此类低阶多项式所需的样本复杂度。我们证明，对于任意回归风险$\eps \in (0,1)$，一个精心设计的具有通道注意力且有限宽度$m \ge Θ({n^4 \log (2n/δ)}/{d^{2\ell_0}})$的双层神经网络，通过标准梯度下降训练，能以$1-δ$的概率（对任意$δ\in (0,1)$）实现最低样本复杂度$n \asymp Θ(d^{\ell_0}/\eps)$；相比之下，代表性样本复杂度为$Θ\pth{d^{\ell_0} \max\set{\eps^{-2},\log d}}$，其中$n$为训练数据量。此外，该样本复杂度不可改进，因为训练所得网络能以至少$1-δ$的概率实现$Θ(d^{\ell_0}/{n})$阶的非参数回归风险锐利速率。另一方面，使用秩为$Θ(d^{\ell_0})$的核函数时，回归风险的极小极大最优速率为$Θ(d^{\ell_0}/{n})$，因此通过梯度下降训练的网络所达到的非参数回归风险速率是极小极大最优的。具有通道注意力的双层神经网络的训练包含两个阶段。在第一阶段，一个可证明的可学习通道选择算法以高概率从第一层激活的初始$L \ge \ell_0$个通道中识别出真实通道数$\ell_0$。这种可学习选择通过对两层参数执行高效的单步梯度下降更新实现，从而为低阶多项式目标实现特征学习。在第二阶段，使用选定通道的激活函数，通过标准梯度下降训练第二层参数。