Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One of these phenomena, Linear Mode Connectivity (LMC), has gained considerable attention due to the intriguing observation that different solutions can be connected by a linear path in the parameter space while maintaining near-constant training and test losses. In this work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which says that the feature maps of every layer in different trained networks are also linearly connected. We provide comprehensive empirical evidence for LLFC across a wide range of settings, demonstrating that whenever two trained networks satisfy LMC (via either spawning or permutation methods), they also satisfy LLFC in nearly all the layers. Furthermore, we delve deeper into the underlying factors contributing to LLFC, which reveal new insights into the spawning and permutation approaches. The study of LLFC transcends and advances our understanding of LMC by adopting a feature-learning perspective.

翻译：近期研究揭示了神经网络训练中许多引人入胜的经验现象，尽管其损失景观和训练动态仍未被充分理解且高度复杂。其中，线性模式连接（LMC）现象因观察到不同解可通过参数空间中的线性路径相连，同时保持训练和测试损失近乎恒定而受到广泛关注。本文提出一种更强的线性连接概念——逐层线性特征连接（LLFC），该概念表明不同训练网络中每一层的特征图也呈线性连接。我们通过涵盖广泛设置的综合经验证据证明了LLFC，表明当两个训练网络满足LMC（通过生成或置换方法实现）时，它们在几乎所有层中也满足LLFC。此外，我们深入探究了LLFC的潜在成因，揭示了对生成与置换方法的新见解。LLFC的研究通过采用特征学习视角，超越了现有对LMC的理解并推动了其发展。