Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eigenvalue of the Hessian of the loss. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in deep matrix factorization/deep linear neural network training problems, resolving an open question posed by Mulayoff & Michaeli (2020). This expression reveals a fundamental property of the loss landscape in deep matrix factorization: Having a constant product of the spectral norms of the left and right intermediate factors across layers is a sufficient condition for flatness. Most notably, in both depth-$2$ matrix and deep overparameterized scalar factorization, we show that this condition is both necessary and sufficient for flatness, which implies that flat minima are spectral-norm balanced even though they are not necessarily Frobenius-norm balanced. To complement our theory, we provide the first empirical characterization of an escape phenomenon during gradient-based training near a minimizer of a deep matrix factorization problem.

翻译：理解损失函数在极小点附近的几何形态，对于解释梯度方法在非凸优化问题（如深度神经网络训练和深度矩阵分解）中的隐式偏好至关重要。刻画该几何形态的一个核心量是损失函数Hessian矩阵的最大特征值。目前，其确切作用尚不明确，因为在一般设置下尚未知晓该锐度度量的精确表达式。本文首次给出了深度矩阵分解/深度线性神经网络训练问题中，任意极小点处平方误差损失Hessian矩阵最大特征值的精确表达式，从而解决了Mulayoff & Michaeli (2020)提出的一个开放性问题。该表达式揭示了深度矩阵分解损失函数景观的一个基本性质：各层中间因子的左右谱范数乘积保持恒定是平坦性的一个充分条件。尤为重要的是，在深度为2的矩阵分解和深度过参数化标量分解中，我们证明该条件既是平坦性的充分条件也是必要条件，这意味着平坦极小点即使不一定是Frobenius范数平衡的，也必然是谱范数平衡的。作为理论的补充，我们首次通过实验刻画了深度矩阵分解问题在极小点附近基于梯度训练过程中的逃逸现象。