Random Feature Model (RFM) with a nonlinear activation function is instrumental in understanding training and generalization performance in high-dimensional learning. While existing research has established an asymptotic equivalence in performance between the RFM and noisy linear models under isotropic data assumptions, empirical observations indicate that the RFM frequently surpasses linear models in practical applications. To address this gap, we ask, "When and how does the RFM outperform linear models?" In practice, inputs often have additional structures that significantly influence learning. Therefore, we explore the RFM under anisotropic input data characterized by spiked covariance in the proportional asymptotic limit, where dimensions diverge jointly while maintaining finite ratios. Our analysis reveals that a high correlation between inputs and labels is a critical factor enabling the RFM to outperform linear models. Moreover, we show that the RFM performs equivalent to noisy polynomial models, where the polynomial degree depends on the strength of the correlation between inputs and labels. Our numerical simulations validate these theoretical insights, confirming the performance-wise superiority of RFM in scenarios characterized by strong input-label correlation.

翻译：具有非线性激活函数的随机特征模型对于理解高维学习中的训练与泛化性能至关重要。现有研究已证明在各项同性数据假设下，随机特征模型与带噪声线性模型在性能上具有渐近等价性，但实证观察表明随机特征模型在实际应用中经常超越线性模型。为弥补这一认知差距，我们提出核心问题："随机特征模型在何种情况下以及如何优于线性模型？" 实践中，输入数据往往具有显著影响学习的附加结构特征。因此，我们在比例渐近极限框架下，研究具有尖峰协方差特性的各向异性输入数据中的随机特征模型，该框架要求维度同步发散且保持有限比率。分析表明：输入与标签之间的高度相关性是促使随机特征模型超越线性模型的关键因素。进一步研究发现，随机特征模型的性能等价于带噪声的多项式模型，其中多项式次数取决于输入与标签之间的相关性强弱。数值模拟验证了这些理论发现，证实了在强输入-标签相关性场景中随机特征模型具有性能优势。