Understanding the properties of well-generalizing minima is at the heart of deep learning research. On the one hand, the generalization of neural networks has been connected to the decision boundary complexity, which is hard to study in the high-dimensional input space. Conversely, the flatness of a minimum has become a controversial proxy for generalization. In this work, we provide the missing link between the two approaches and show that the Hessian top eigenvectors characterize the decision boundary learned by the neural network. Notably, the number of outliers in the Hessian spectrum is proportional to the complexity of the decision boundary. Based on this finding, we provide a new and straightforward approach to studying the complexity of a high-dimensional decision boundary; show that this connection naturally inspires a new generalization measure; and finally, we develop a novel margin estimation technique which, in combination with the generalization measure, precisely identifies minima with simple wide-margin boundaries. Overall, this analysis establishes the connection between the Hessian and the decision boundary and provides a new method to identify minima with simple wide-margin decision boundaries.

翻译：理解良好泛化的极小值特性是深度学习研究的核心问题。一方面，神经网络的泛化能力与决策边界的复杂度相关联，但在高维输入空间中研究这一复杂度极具挑战性；另一方面，极小值的平坦度作为泛化能力的替代指标一直存在争议。本研究填补了两种研究路径之间的缺失环节，证明海森矩阵的顶特征向量刻画了神经网络习得的决策边界特征。值得注意的是，海森矩阵谱中的异常值数量与决策边界复杂度成正比。基于此发现，我们提出了一种研究高维决策边界复杂度的全新简化方法；揭示了这一内在联系如何自然衍生出新的泛化度量指标；最终开发了创新的边界估计技术，该技术结合泛化度量指标能够精确识别具有简单宽间隔边界的极小值点。整体而言，本研究建立了海森矩阵与决策边界之间的理论关联，并提供了识别具有简单宽间隔决策边界极小值的新方法。