Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One of these phenomena, Linear Mode Connectivity (LMC), has gained considerable attention due to the intriguing observation that different solutions can be connected by a linear path in the parameter space while maintaining near-constant training and test losses. In this work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which says that the feature maps of every layer in different trained networks are also linearly connected. We provide comprehensive empirical evidence for LLFC across a wide range of settings, demonstrating that whenever two trained networks satisfy LMC (via either spawning or permutation methods), they also satisfy LLFC in nearly all the layers. Furthermore, we delve deeper into the underlying factors contributing to LLFC, which reveal new insights into the spawning and permutation approaches. The study of LLFC transcends and advances our understanding of LMC by adopting a feature-learning perspective.

翻译：近期研究揭示了神经网络训练中诸多引人入胜的经验现象，尽管其损失景观与训练动态仍未被充分理解且高度复杂。其中，线性模式连接（LMC）现象因展现出不同解可通过参数空间中的线性路径相连，且训练与测试损失近乎保持不变，而备受关注。本文提出一种更强的线性连接概念——逐层线性特征连接（LLFC），即不同训练网络中每一层的特征映射也呈线性连接。我们通过广泛设置下的综合经验证据表明，当两个训练网络满足LMC（通过分支或排列方法达成）时，它们几乎在所有层中也满足LLFC。此外，我们深入探究了促成LLFC的潜在因素，揭示了分支与排列方法的新见解。对LLFC的研究从特征学习视角超越并深化了我们对LMC的理解。