Stochastic Gradient Descent (SGD), a widely used optimization algorithm in deep learning, is often limited to converging to local optima due to the non-convex nature of the problem. Leveraging these local optima to improve model performance remains a challenging task. Given the inherent complexity of neural networks, the simple arithmetic averaging of the obtained local optima models in undesirable results. This paper proposes a {\em soft merging} method that facilitates rapid merging of multiple models, simplifies the merging of specific parts of neural networks, and enhances robustness against malicious models with extreme values. This is achieved by learning gate parameters through a surrogate of the $l_0$ norm using hard concrete distribution without modifying the model weights of the given local optima models. This merging process not only enhances the model performance by converging to a better local optimum, but also minimizes computational costs, offering an efficient and explicit learning process integrated with stochastic gradient descent. Thorough experiments underscore the effectiveness and superior performance of the merged neural networks.

翻译：随机梯度下降（SGD）作为深度学习广泛使用的优化算法，由于问题的非凸性往往局限于收敛至局部最优解。如何利用这些局部最优解提升模型性能仍是一项具有挑战性的任务。鉴于神经网络的内在复杂性，对所得局部最优模型进行简单算术平均会产生不良结果。本文提出一种"软合并"方法，该方法能够快速合并多个模型，简化神经网络特定部分的合并过程，并增强对含极端值的恶意模型的鲁棒性。这一目标通过使用硬直方分布（hard concrete distribution）作为$l_0$范数的代理来学习门控参数实现，且无需修改给定局部最优模型的权重。该合并过程不仅通过收敛至更优局部最优点提升了模型性能，还最大限度降低了计算成本，为随机梯度下降提供了高效且显式的学习流程。充分的实验验证了合并后神经网络的有效性与优越性能。