Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.

翻译：探索损失景观有助于理解深度神经网络的内在原理。近期研究提出，在平坦谷与尖锐谷之外，还存在一种额外的谷非对称性，但尚未深入探究其成因或影响。我们的研究系统性地考察了影响深度神经网络谷对称性的因素，包括：(1) 影响收敛点的数据集、网络架构、初始化和超参数；(2) 用于一维可视化的噪声幅度与方向。主要发现表明，噪声与收敛点之间的符号一致性程度是谷对称性的关键指标。从ReLU激活函数和softmax函数角度的理论见解可解释这一有趣现象。我们的发现推动了模型融合场景中的新理解与应用：(1) 不同模型插值的有效性与其符号一致性比率显著相关；(2) 在联邦学习中施加符号对齐成为模型参数对齐的创新方法。