The elusive nature of gradient-based optimization in neural networks is tied to their loss landscape geometry, which is poorly understood. However recent work has brought solid evidence that there is essentially no loss barrier between the local solutions of gradient descent, once accounting for weight-permutations that leave the network's computation unchanged. This raises questions for approximate inference in Bayesian neural networks (BNNs), where we are interested in marginalizing over multiple points in the loss landscape. In this work, we first extend the formalism of marginalized loss barrier and solution interpolation to BNNs, before proposing a matching algorithm to search for linearly connected solutions. This is achieved by aligning the distributions of two independent approximate Bayesian solutions with respect to permutation matrices. We build on the results of Ainsworth et al. (2023), reframing the problem as a combinatorial optimization one, using an approximation to the sum of bilinear assignment problem. We then experiment on a variety of architectures and datasets, finding nearly zero marginalized loss barriers for linearly connected solutions.

翻译：神经网络中基于梯度的优化难以捉摸的特性与其损失景观几何结构密切相关，而后者目前尚未被充分理解。然而近期研究提供了有力证据表明，在考虑保持网络计算不变性的权重置换后，梯度下降的局部解之间本质上不存在损失壁垒。这一发现对贝叶斯神经网络（BNN）中的近似推断提出了新问题——我们关注的是对损失景观中多个点的边际化。本研究首先将边际化损失壁垒和解插值的形式化体系扩展至BNNs，随后提出一种用于搜索线性连通解的匹配算法。该算法通过基于置换矩阵对齐两个独立近似贝叶斯解的分布实现。我们借鉴Ainsworth等人(2023)的研究成果，将问题重构为组合优化问题，采用双线性指派问题的和近似求解。最终在多种架构和数据集上的实验表明，线性连通解具有近乎为零的边际化损失壁垒。