We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.

翻译：我们利用师生框架研究了特征叠加在幂律训练动态涌现中的作用。首先推导了无叠加情况下的解析理论，证明幂律训练指数同时依赖于输入数据统计特性与通道重要性。值得注意的是，我们发现叠加瓶颈会引发向普适幂律指数$\sim 1$的转变，该指数独立于数据与通道统计特性。这种叠加条件下的"随时间倒数"训练模式，相较于无叠加时的纯顺序学习，实现了高达十倍的加速。叠加导致训练加速且产生数据无关幂律指数的发现，可能对包括生产级大语言模型在内广泛采用叠加机制的神经网络具有重要启示。