In recent years, significant attention in deep learning theory has been devoted to analyzing the generalization performance of models with multiple layers of Gaussian random features. However, few works have considered the effect of feature anisotropy; most assume that features are generated using independent and identically distributed Gaussian weights. Here, we derive learning curves for models with many layers of structured Gaussian features. We show that allowing correlations between the rows of the first layer of features can aid generalization, while structure in later layers is generally detrimental. Our results shed light on how weight structure affects generalization in a simple class of solvable models.

翻译：近年来，深度学习理论领域的诸多研究聚焦于分析多层高斯随机特征模型的泛化性能。然而，少有工作考虑特征各向异性的影响：大多数研究假设特征是通过独立同分布的高斯权重生成的。本文推导了多层结构化高斯特征模型的学习曲线。研究发现，允许第一层特征行之间存在相关性有助于提升泛化能力，而后续层中的结构则通常会产生不利影响。我们的结果揭示了在一个简易可解模型类别中，权重结构如何影响泛化性能。