We present a new approach to understanding the relationship between loss curvature and input-output model behaviour in deep learning. Specifically, we use existing empirical analyses of the spectrum of deep network loss Hessians to ground an ansatz tying together the loss Hessian and the input-output Jacobian of a deep neural network over training samples throughout training. We then prove a series of theoretical results which quantify the degree to which the input-output Jacobian of a model approximates its Lipschitz norm over a data distribution, and deduce a novel generalisation bound in terms of the empirical Jacobian. We use our ansatz, together with our theoretical results, to give a new account of the recently observed progressive sharpening phenomenon, as well as the generalisation properties of flat minima. Experimental evidence is provided to validate our claims.

翻译：我们提出了一种理解深度学习中损失曲率与输入-输出模型行为之间关系的新方法。具体而言，我们利用先前对深度网络损失Hessian矩阵谱的实证分析，建立了一个将损失Hessian矩阵与深度神经网络在训练样本上（贯穿整个训练过程）的输入-输出雅可比矩阵联系起来的假设。随后，我们证明了一系列理论结果，量化了模型输入-输出雅可比矩阵在数据分布上逼近其Lipschitz范数的程度，并基于经验雅可比矩阵推导出一个新的泛化界。结合这一假设与我们的理论结果，我们为近期观察到的渐进尖锐化现象以及平坦极小值的泛化特性提供了新的解释。我们提供了实验证据来验证我们的论断。