We study the problem of gradient descent learning of a single-index target function $f_*(\boldsymbol{x}) = \textstyle\sigma_*\left(\langle\boldsymbol{x},\boldsymbol{\theta}\rangle\right)$ under isotropic Gaussian data in $\mathbb{R}^d$, where the link function $\sigma_*:\mathbb{R}\to\mathbb{R}$ is an unknown degree $q$ polynomial with information exponent $p$ (defined as the lowest degree in the Hermite expansion). Prior works showed that gradient-based training of neural networks can learn this target with $n\gtrsim d^{\Theta(p)}$ samples, and such statistical complexity is predicted to be necessary by the correlational statistical query lower bound. Surprisingly, we prove that a two-layer neural network optimized by an SGD-based algorithm learns $f_*$ of arbitrary polynomial link function with a sample and runtime complexity of $n \asymp T \asymp C(q) \cdot d\mathrm{polylog} d$, where constant $C(q)$ only depends on the degree of $\sigma_*$, regardless of information exponent; this dimension dependence matches the information theoretic limit up to polylogarithmic factors. Core to our analysis is the reuse of minibatch in the gradient computation, which gives rise to higher-order information beyond correlational queries.

翻译：我们研究在$\mathbb{R}^d$中各向同性高斯数据下，梯度下降学习单索引目标函数$f_*(\boldsymbol{x}) = \textstyle\sigma_*\left(\langle\boldsymbol{x},\boldsymbol{\theta}\rangle\right)$的问题，其中链接函数$\sigma_*:\mathbb{R}\to\mathbb{R}$是信息指数为$p$（定义为埃尔米特展开中的最低次数）的未知$q$次多项式。先前研究表明，基于梯度的神经网络训练能够以$n\gtrsim d^{\Theta(p)}$的样本量学习该目标，且相关统计查询下界预测这种统计复杂度是必要的。令人惊讶的是，我们证明通过基于SGD的算法优化的两层神经网络，能够以$n \asymp T \asymp C(q) \cdot d\mathrm{polylog} d$的样本和计算复杂度学习任意多项式链接函数的$f_*$，其中常数$C(q)$仅取决于$\sigma_*$的次数，而与信息指数无关；该维度依赖关系与信息论极限的差异仅为多对数因子。我们分析的核心在于梯度计算中对小批量的重用，这产生了超越相关查询的高阶信息。