Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity,and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.

翻译：理解深度神经网络如何从数据中学习有用的内部表征，仍是深度学习理论中的核心开放问题。我们提出神经低度滤波（Neural LoFi），一种基于梯度训练的风格化极限，在该极限下层级特征学习成为显式的迭代谱过程。在此极限中，每层动力学解耦：给定当前表征后，下一层选择与标签具有最大可及低度相关性的方向。这为深度学习提供了可处理的替代机制，并赋予其自然的核空间解释。Neural LoFi为研究惰性区域之外的层层特征学习提供了数学明确的框架。它预测了表征如何逐层选择，解释了概念涌现如何随特定样本复杂度产生，并给出了深度通过低度组合性逐步从旧特征构建新特征的具体机制。我们通过全连接和卷积架构的机制性实验补充理论，证明Neural LoFi优于惰性随机特征基线、恢复有意义的结构化滤波器，并能用真实数据集预测与早期梯度下降特征发现一致的表征。