Neural networks (NNs) are central to modern machine learning and achieve state-of-the-art results in many applications. However, the relationship between loss geometry and generalization is still not well understood. The local geometry of the loss function near a critical point is well-approximated by its quadratic form, obtained through a second-order Taylor expansion. The coefficients of the quadratic term correspond to the Hessian matrix, whose eigenspectrum allows us to evaluate the sharpness of the loss at the critical point. Extensive research suggests flat critical points generalize better, while sharp ones lead to higher generalization error. However, sharpness requires the Hessian eigenspectrum, but general matrix characteristic equations have no closed-form solution. Therefore, most existing studies on evaluating loss sharpness rely on numerical approximation methods. Existing closed-form analyses of the eigenspectrum are primarily limited to simplified architectures, such as linear or ReLU-activated networks; consequently, theoretical analysis of smooth nonlinear multilayer neural networks remains limited. Against this background, this study focuses on nonlinear, smooth multilayer neural networks and derives a closed-form upper bound for the maximum eigenvalue of the Hessian with respect to the cross-entropy loss by leveraging the Wolkowicz-Styan bound. Specifically, the derived upper bound is expressed as a function of the affine transformation parameters, hidden layer dimensions, and the degree of orthogonality among the training samples. The primary contribution of this paper is an analytical characterization of loss sharpness in smooth nonlinear multilayer neural networks via a closed-form expression, avoiding explicit numerical eigenspectrum computation. We hope that this work provides a small yet meaningful step toward unraveling the mysteries of deep learning.

翻译：[translated abstract in Chinese] 神经网络是现代机器学习的核心，在众多应用中取得了最先进的成果。然而，损失几何与泛化性能之间的关系仍未得到充分理解。临界点附近损失函数的局部几何形状可通过其二阶泰勒展开得到的二次形式精确逼近，其中二次项系数对应海森矩阵，其特征谱可用于评估临界点处损失的尖锐度。大量研究表明，平坦临界点具有更好的泛化能力，而尖锐临界点则会导致更高的泛化误差。然而，尖锐度分析需要海森特征谱，但一般矩阵特征方程不存在闭式解。因此，现有大多数损失尖锐度评估研究依赖数值近似方法。现有特征谱闭式分析主要局限于简化架构（如线性或ReLU激活网络），导致光滑非线性多层神经网络的理论分析仍十分有限。在此背景下，本研究聚焦于非线性光滑多层神经网络，通过利用沃科维茨-斯蒂安界推导出交叉熵损失下海森矩阵最大特征值的闭式上界。具体而言，该上界可表示为仿射变换参数、隐藏层维度以及训练样本间正交度函数的形式。本文的主要贡献在于通过闭式表达式对光滑非线性多层神经网络的损失尖锐度进行理论描述，避免了显式数值特征谱计算。希望本工作能为破解深度学习奥秘提供微小而有意义的进展。