In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity could exhibit a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights -- in particular their effective rank -- influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting.

翻译：在理论神经科学中，近期研究借助深度学习工具探索了网络属性如何关键性地影响其学习动态。值得注意的是，初始权重分布方差较小（或较大）可能分别产生丰富（或懒惰）学习状态，其中网络状态与表征在学习过程中会发生显著（或微小）变化。然而在生物学中，神经回路连接可能呈现低秩结构，这与上述研究通常采用的随机初始化方式存在显著差异。因此，本文探究了初始权重结构——特别是其有效秩——如何影响网络的学习状态。通过实证与理论分析，我们发现高秩初始化通常会产生表征较小网络变化的懒惰学习，这一结论在使用实验驱动初始连接的递归神经网络中也得到验证。相反，低秩初始化则使学习偏向于更丰富的状态。但值得注意的是，作为该规律的例外情况，我们发现当低秩初始化与任务及数据统计特征对齐时，仍可能产生懒惰学习。本研究揭示了初始权重结构在塑造学习状态中的核心作用，对塑性的代谢成本与灾难性遗忘风险具有重要启示。