The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the earlier work by Luo et al.~\cite{luo2021phase}, we present a phase diagram of initial condensation for two-layer neural networks. Condensation is a phenomenon wherein the weight vectors of neural networks concentrate on isolated orientations during the training process, and it is a feature in non-linear learning process that enables neural networks to possess better generalization abilities. Our phase diagram serves to provide a comprehensive understanding of the dynamical regimes of neural networks and their dependence on the choice of hyperparameters related to initialization. Furthermore, we demonstrate in detail the underlying mechanisms by which small initialization leads to condensation at the initial training stage.

翻译：神经网络在初始化尺度变化时表现出不同行为的现象，至今仍是深度学习研究中的一个谜团。本文基于Luo等人~\cite{luo2021phase}的前期工作，提出了两层神经网络初始凝聚的相图。凝聚是指神经网络权重向量在训练过程中集中到孤立方向的现象，这是非线性学习过程中的一个特征，能够使神经网络具备更强的泛化能力。我们的相图旨在全面理解神经网络的动力学机制及其对初始化相关超参数选择的依赖性。此外，我们详细阐明了小初始化导致训练初期凝聚的内在机理。