Neural networks excel at discovering statistical patterns in high-dimensional data sets. In practice, higher-order cumulants, which quantify the non-Gaussian correlations between three or more variables, are particularly important for the performance of neural networks. But how efficient are neural networks at extracting features from higher-order cumulants? We study this question in the spiked cumulant model, where the statistician needs to recover a privileged direction or "spike" from the order-$p\ge 4$ cumulants of $d$-dimensional inputs. We first characterise the fundamental statistical and computational limits of recovering the spike by analysing the number of samples $n$ required to strongly distinguish between inputs from the spiked cumulant model and isotropic Gaussian inputs. We find that statistical distinguishability requires $n\gtrsim d$ samples, while distinguishing the two distributions in polynomial time requires $n \gtrsim d^2$ samples for a wide class of algorithms, i.e. those covered by the low-degree conjecture. These results suggest the existence of a wide statistical-to-computational gap in this problem. Numerical experiments show that neural networks learn to distinguish the two distributions with quadratic sample complexity, while "lazy" methods like random features are not better than random guessing in this regime. Our results show that neural networks extract information from higher-order correlations in the spiked cumulant model efficiently, and reveal a large gap in the amount of data required by neural networks and random features to learn from higher-order cumulants.

翻译：神经网络擅长发现高维数据集中的统计模式。实践中，高阶累积量（用于量化三个或更多变量之间非高斯相关性的统计量）对神经网络的性能尤为重要。但神经网络从高阶累积量中提取特征的效率如何？我们在尖峰累积量模型下研究这一问题，其中统计学家需要从d维输入的四阶及以上累积量中恢复一个特权方向（即"尖峰"）。我们首先通过分析区分尖峰累积量模型输入与各向同性高斯输入所需的最小样本量n，刻画恢复该尖峰的基本统计极限与计算极限。研究发现，统计可区分性需要n≳d个样本，而在一大类算法（即低阶猜想涵盖的算法）中，多项式时间内区分两种分布需要n≳d²个样本。这表明该问题存在显著的统计-计算鸿沟。数值实验表明，神经网络能以二次样本复杂度学习区分两种分布，而随机特征等"惰性"方法在此场景下表现不优于随机猜测。我们的研究表明，神经网络能高效地从尖峰累积量模型的高阶相关性中提取信息，并揭示了神经网络与随机特征学习高阶累积量所需数据量之间的巨大差异。