We study the problem of learning a low-degree spherical polynomial of degree $k_0 = Θ(1) \ge 1$ defined on the unit sphere in $\RR^d$ by training an over-parameterized two-layer neural network with augmented feature in this paper. Our main result is the significantly improved sample complexity for learning such low-degree polynomials. We show that, for any regression risk $\eps \in (0, Θ(d^{-k_0})]$, an over-parameterized two-layer neural network trained by a novel Gradient Descent with Projection (GDP) requires a sample complexity of $n \asymp Θ( \log(4/δ) \cdot d^{k_0}/\eps)$ with probability $1-δ$ for $δ\in (0,1)$, in contrast with the representative sample complexity $Θ(d^{k_0} \max\set{\eps^{-2},\log d})$. Moreover, such sample complexity is nearly unimprovable since the trained network renders a nearly optimal rate of the nonparametric regression risk of the order $\log({4}/δ) \cdot Θ(d^{k_0}/{n})$ with probability at least $1-δ$. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank $Θ(d^{k_0})$ is $Θ(d^{k_0}/{n})$, so that the rate of the nonparametric regression risk of the network trained by GDP is nearly minimax optimal. In the case that the ground truth degree $k_0$ is unknown, we present a novel and provable adaptive degree selection algorithm which identifies the true degree and achieves the same nearly optimal regression rate. To the best of our knowledge, this is the first time that a nearly optimal risk bound is obtained by training an over-parameterized neural network with a popular activation function (ReLU) and algorithmic guarantee for learning low-degree spherical polynomials. Due to the feature learning capability of GDP, our results are beyond the regular Neural Tangent Kernel (NTK) limit.

翻译：本文研究通过训练具有增强特征的过参数化双层神经网络，学习定义在 $\RR^d$ 单位球面上的低次球面多项式（次数 $k_0 = Θ(1) \ge 1$）的问题。主要成果是显著降低了学习此类低次多项式的样本复杂度。我们证明：对于任意回归风险 $\eps \in (0, Θ(d^{-k_0})]$，当 $\delta \in (0,1)$ 时，使用新型梯度下降投影法（GDP）训练的过参数化双层神经网络，在 $1-\delta$ 概率下需要样本复杂度 $n \asymp Θ( \log(4/δ) \cdot d^{k_0}/\eps)$，而代表性样本复杂度为 $Θ(d^{k_0} \max\set{\eps^{-2},\log d})$。此外，该样本复杂度几乎不可改进，因为训练后的网络以至少 $1-\delta$ 概率达到非参数回归风险 $\log({4}/δ) \cdot Θ(d^{k_0}/{n})$ 量级的近最优速率。另一方面，秩为 $Θ(d^{k_0})$ 的核函数的回归风险极小极大最优速率为 $Θ(d^{k_0}/{n})$，故经 GDP 训练的网络的非参数回归风险速率接近极小极大最优。当真实次数 $k_0$ 未知时，我们提出一种新颖且可证明的自适应次数选择算法，该算法可识别真实次数并实现相同的近最优回归速率。据我们所知，这是首次通过训练具有常用激活函数（ReLU）的过参数化神经网络，在算法保证下实现学习低次球面多项式的近最优风险界。由于 GDP 具备特征学习能力，我们的结果超越了传统神经切线核（NTK）极限。