In recent years, significant attention in deep learning theory has been devoted to analyzing the generalization performance of models with multiple layers of Gaussian random features. However, few works have considered the effect of feature anisotropy; most assume that features are generated using independent and identically distributed Gaussian weights. Here, we derive learning curves for models with many layers of structured Gaussian features. We show that allowing correlations between the rows of the first layer of features can aid generalization, while structure in later layers is generally detrimental. Our results shed light on how weight structure affects generalization in a simple class of solvable models.

翻译：近年来，深度学习理论中大量研究致力于分析多层高斯随机特征模型的泛化性能。然而，鲜有工作考虑特征各向异性的影响；多数研究假设特征是通过独立同分布的高斯权重生成的。本文推导了具有多层结构化高斯特征模型的学习曲线。我们证明，允许第一层特征行间存在相关性有助于提升泛化能力，而后续层中的结构通常会产生不利影响。我们的研究揭示了在简单可解模型类别中，权重结构如何影响泛化性能。