Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eigenvalue of the Hessian of the loss. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in deep matrix factorization/deep linear neural network training problems, resolving an open question posed by Mulayoff & Michaeli (2020). This expression reveals a fundamental property of the loss landscape in deep matrix factorization: Having a constant product of the spectral norms of the left and right intermediate factors across layers is a sufficient condition for flatness. Most notably, in both depth-$2$ matrix and deep overparameterized scalar factorization, we show that this condition is both necessary and sufficient for flatness, which implies that flat minima are spectral-norm balanced even though they are not necessarily Frobenius-norm balanced. To complement our theory, we provide the first empirical characterization of an escape phenomenon during gradient-based training near a minimizer of a deep matrix factorization problem.

翻译：理解损失函数在极小值附近的几何形态，对于解释梯度方法在非凸优化问题（如深度神经网络训练和深度矩阵分解）中的隐式偏好至关重要。刻画此几何形态的一个核心量是损失函数海森矩阵的最大特征值。目前，其精确作用尚不明确，因为在一般设置下尚无该锐度度量的精确表达式。本文首次给出了深度矩阵分解/深度线性神经网络训练问题中，任意极小值点处平方误差损失海森矩阵最大特征值的精确表达式，从而解决了 Mulayoff & Michaeli (2020) 提出的一个开放性问题。该表达式揭示了深度矩阵分解中损失函数的一个基本性质：各层中间因子的左、右矩阵谱范数之积为常数是平坦性的一个充分条件。特别值得注意的是，在深度为 2 的矩阵分解以及深度过参数化标量分解中，我们证明该条件既是平坦性的充分条件也是必要条件，这意味着平坦极小值点即使不一定是 Frobenius 范数平衡的，也必然是谱范数平衡的。作为理论的补充，我们首次通过实验刻画了深度矩阵分解问题在极小值点附近进行梯度训练时的一种逃逸现象。