Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.

翻译：关键学习期是发育早期的一段时期，在此期间暂时的感觉缺陷可能对行为和习得的表征产生永久性影响。尽管生物网络与人工网络存在根本性差异，但在这两种系统中都通过实证观察到了关键学习期的存在。这表明关键期可能是学习的基本属性，而非生物学的偶然现象。然而，关键期为何会在深度网络中涌现仍是一个悬而未决的问题，特别是尚不清楚在这两种系统中观察到的关键期是否依赖于特定的架构或优化细节。为分离关键影响因素，我们聚焦于深度线性网络模型，并证明这类网络在具备可解析处理特性的同时，竟也展现出与生物网络及人工网络相似的行为特征。我们证明关键期的存在取决于模型的深度及数据分布的结构。通过解析推导与仿真实验，我们还证明了特征学习与不同信号源之间的竞争密切相关。最后，我们将分析拓展至多任务学习场景，证明在某些任务上的预训练可能损害新任务的迁移性能，并揭示了这种影响如何取决于任务间的关联性及预训练阶段的持续时间。据我们所知，本研究首次提出了可解析处理的模型，为理解关键学习期在生物与人工网络中涌现的机制提供了理论依据。