We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. Interfering with the learning process during this initial stage can permanently impair the development of a skill, both in artificial and biological systems where the phenomenon is known as a critical learning period. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations. This evidence challenges the view, engendered by analysis of wide and shallow networks, that early learning dynamics of neural networks are simple, akin to those of a linear model. Indeed, we show that even deep linear networks exhibit critical learning periods for multi-source integration, while shallow networks do not. To better understand how the internal representations change according to disturbances or sensory deficits, we introduce a new measure of source sensitivity, which allows us to track the inhibition and integration of sources during training. Our analysis of inhibition suggests cross-source reconstruction as a natural auxiliary training objective, and indeed we show that architectures trained with cross-sensor reconstruction objectives are remarkably more resilient to critical periods. Our findings suggest that the recent success in self-supervised multi-modal training compared to previous supervised efforts may be in part due to more robust learning dynamics and not solely due to better architectures and/or more data.

翻译：我们研究表明，神经网络整合多源信息的能力关键取决于训练初期是否接触到适当相关的信号。在初始阶段干扰学习过程可能永久损害技能发展——这种现象在人工系统与生物系统中均被称为关键学习期。我们发现，关键期产生于复杂且不稳定的早期暂态动力学过程，这一过程决定了训练系统的最终性能及其学习表征。该发现挑战了由宽浅网络分析得出的观点——即神经网络早期学习动态类似于线性模型且简单。事实上，我们证明即使深度线性网络在多源整合中也表现出关键学习期，而浅层网络则不会。为更好理解内部表征如何随扰动或感觉缺陷变化，我们引入新的源敏感性度量方法，可追踪训练过程中对信息源的抑制与整合。对抑制机制的分析表明，跨源重建可作为自然的辅助训练目标，且实验证实采用跨传感器重建目标训练的架构对关键期具有显著更强的鲁棒性。我们的发现提示，近期自监督多模态训练相比先前监督方法的成功，可能部分源于更稳健的学习动态，而不仅仅是更好的架构和/或更多数据。