Averaging neural network parameters is an intuitive method for fusing the knowledge of two independent models. It is most prominently used in federated learning. If models are averaged at the end of training, this can only lead to a good performing model if the loss surface of interest is very particular, i.e., the loss in the midpoint between the two models needs to be sufficiently low. This is impossible to guarantee for the non-convex losses of state-of-the-art networks. For averaging models trained on vastly different datasets, it was proposed to average only the parameters of particular layers or combinations of layers, resulting in better performing models. To get a better understanding of the effect of layer-wise averaging, we analyse the performance of the models that result from averaging single layers, or groups of layers. Based on our empirical and theoretical investigation, we introduce a novel notion of the layer-wise linear connectivity, and show that deep networks do not have layer-wise barriers between them. In addition, we analyze layer-wise personalization averaging and conjecture that in particular problem setup all partial aggregations result in the approximately same performance.

翻译：将神经网络参数取平均是一种融合两个独立模型知识的直观方法，该技术最常用于联邦学习。若在训练结束时对模型取平均，只有当损失曲面具有特殊性质（即两个模型中间点的损失值足够低）时，才能获得性能良好的模型。对于当前先进网络的非凸损失函数而言，这一条件无法保证。针对在不同数据集上训练的模型取平均问题，已有研究提出仅对特定层或层组合的参数进行平均，从而获得更优性能的模型。为深入理解分层平均的影响，我们分析了单层或层组参数平均后所得模型的性能。基于理论与实验研究，我们提出分层线性连通性的新概念，并证明深层网络之间不存在分层级障碍。此外，我们分析了分层个性化平均方法，并推测在特定问题设置下，所有部分聚合方法将产生近似相同的性能。