We study a prototypical situation when a learned predictor can discover useful low-dimensional structure in data, while using fewer samples than are needed for accurate prediction. Specifically, we consider the problem of recovering a multi-index polynomial $f^*(x)=h(Ux)$, with $U\in\mathbb{R}^{r\times d}$ and $r\ll d$, from finitely many data/label pairs. Importantly, the target function depends on input $x$ only through the projection onto an unknown $r$-dimensional central subspace. The algorithm we analyze is appealingly simple: fit kernel ridge regression (KRR) to the data and compute the Average Gradient Outer Product (AGOP) from the fitted predictor. Our main results show that under reasonable assumptions the top $r$-dimensional eigenspace of AGOP provably recovers the central subspace, even in regimes when the prediction error remains large. Specifically, if the target function $f^*$ has degree $p^*$, it is known that $n\asymp d^{p^*}$ samples are necessary for KRR to achieve accurate prediction. In contrast, we show that if a low degree $p$ component of $f^*$ already carries all relevant directions for prediction, subspace recovery occurs in the much lower sample regime $n\asymp d^{p+δ}$ for any $δ\in(0,1)$. Our results thus demonstrate a separation between prediction and representation, and provide an explanation for why iterative kernel methods such as Recursive Feature Machines (RFM) can be sample-efficient in practice.

翻译：我们研究了一种典型情况，即学习到的预测器能够在数据中发现有用的低维结构，同时使用的样本数少于精确预测所需。具体而言，我们考虑从有限的数据/标签对中恢复多指标多项式 $f^*(x)=h(Ux)$（其中 $U\in\mathbb{R}^{r\times d}$，$r\ll d$）的问题。重要的是，目标函数仅通过投影到未知的 $r$ 维中心子空间依赖于输入 $x$。我们分析的算法简单而有吸引力：对数据拟合核岭回归（KRR），并从拟合的预测器中计算平均梯度外积（AGOP）。我们的主要结果表明，在合理假设下，AGOP 的前 $r$ 维特征空间可以证明恢复中心子空间，即使在预测误差仍然很大的情况下也是如此。具体而言，如果目标函数 $f^*$ 具有次数 $p^*$，则已知 KRR 需要 $n\asymp d^{p^*}$ 个样本才能实现精确预测。相比之下，我们证明如果 $f^*$ 的低次数 $p$ 分量已包含所有与预测相关的方向，则在样本量较低的区域 $n\asymp d^{p+δ}$（其中 $δ\in(0,1)$）中，子空间恢复即可实现。因此，我们的结果展示了预测与表示之间的分离，并为递归特征机（RFM）等迭代核方法在实践中具有样本高效性提供了解释。