We explore element-wise convex combinations of two permutation-aligned neural network parameter vectors $\Theta_A$ and $\Theta_B$ of size $d$. We conduct extensive experiments by examining various distributions of such model combinations parametrized by elements of the hypercube $[0,1]^{d}$ and its vicinity. Our findings reveal that broad regions of the hypercube form surfaces of low loss values, indicating that the notion of linear mode connectivity extends to a more general phenomenon which we call mode combinability. We also make several novel observations regarding linear mode connectivity and model re-basin. We demonstrate a transitivity property: two models re-based to a common third model are also linear mode connected, and a robustness property: even with significant perturbations of the neuron matchings the resulting combinations continue to form a working model. Moreover, we analyze the functional and weight similarity of model combinations and show that such combinations are non-vacuous in the sense that there are significant functional differences between the resulting models.

翻译：我们探索了两个大小为$d$的排列对齐神经网络参数向量$\Theta_A$和$\Theta_B$的元素级凸组合。通过研究以超立方体$[0,1]^{d}$及其邻近区域元素为参数的各种模型组合分布，我们开展了大量实验。实验结果表明，超立方体的广阔区域形成了低损失值的曲面，这说明线性模式连通性可推广至一种我们称之为模式可组合性的更普遍现象。我们还针对线性模式连通性和模型重盆地提出了若干新发现。我们证明了传递性：两个与同一第三个模型重对齐的模型同样线性模式连通；以及鲁棒性：即使神经元匹配受到显著扰动，所得组合仍能构成有效模型。此外，我们分析了模型组合的功能与权重相似性，并表明此类组合并非空泛——组合所得模型之间存在显著的功能差异。