When a neural network can learn multiple distinct algorithms to solve a task, how does it "choose" between them during training? To approach this question, we take inspiration from ecology: when multiple species coexist, they eventually reach an equilibrium where some survive while others die out. Analogously, we suggest that a neural network at initialization contains many solutions (representations and algorithms), which compete with each other under pressure from resource constraints, with the "fittest" ultimately prevailing. To investigate this Survival of the Fittest hypothesis, we conduct a case study on neural networks performing modular addition, and find that these networks' multiple circular representations at different Fourier frequencies undergo such competitive dynamics, with only a few circles surviving at the end. We find that the frequencies with high initial signals and gradients, the "fittest," are more likely to survive. By increasing the embedding dimension, we also observe more surviving frequencies. Inspired by the Lotka-Volterra equations describing the dynamics between species, we find that the dynamics of the circles can be nicely characterized by a set of linear differential equations. Our results with modular addition show that it is possible to decompose complicated representations into simpler components, along with their basic interactions, to offer insight on the training dynamics of representations.

翻译：当神经网络能够学习多种不同算法来解决同一任务时，它在训练过程中如何在这些算法之间进行“选择”？为探讨这一问题，我们从生态学中获得启发：当多个物种共存时，它们最终会达到某种平衡状态，其中部分物种存活而其他物种消亡。类似地，我们认为神经网络在初始化时包含多种解决方案（表征与算法），这些方案在资源约束的压力下相互竞争，最终“最适者”得以胜出。为验证这一“适者生存”假说，我们对执行模加法的神经网络进行了案例研究，发现这些网络中不同傅里叶频率下的多重圆形表征确实经历了此类竞争动态，最终仅有少数圆形表征得以保留。研究发现，具有高初始信号与梯度的频率——即“最适者”——更有可能存活。通过增加嵌入维度，我们还观察到更多存活频率。受描述物种间动态的Lotka-Volterra方程启发，我们发现圆形表征的动态可以通过一组线性微分方程很好地刻画。我们在模加法任务中的结果表明，将复杂表征分解为更简单的组件及其基本相互作用是可行的，这为理解表征的训练动态提供了新的视角。