In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity generally has a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights, in particular their effective rank, influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting.

翻译：在理论神经科学领域，近期研究借助深度学习工具探索网络属性如何关键影响其学习动态。值得注意的是，初始权重分布的小方差（或大方差）可能产生丰富（或惰性）学习模式，表现为学习过程中网络状态和表征发生显著（或微小）变化。然而，生物神经回路连接通常具有低秩结构，这与该类研究中普遍采用的随机初始化方式存在本质差异。为此，本文探究初始权重结构（特别是其有效秩）如何影响网络学习模式。通过实证与理论分析，我们发现高秩初始化通常导致更小的网络变化，表征更惰性的学习——这一结论也在基于实验驱动的循环神经网络连接矩阵中得到验证。相反，低秩初始化使学习偏向丰富模式。但值得注意的是，存在一个关键例外：当低秩初始化与任务及数据统计特性对齐时，仍可能发生惰性学习。本研究揭示了初始权重结构在塑造学习模式中的核心作用，对突触可塑性能量代谢成本与灾难性遗忘风险具有重要启示。