One of the fundamental problems in machine learning is generalization. In neural network models with a large number of weights (parameters), many solutions can be found to fit the training data equally well. The key question is which solution can describe testing data not in the training set. Here, we report the discovery of an exact duality (equivalence) between changes in activities in a given layer of neurons and changes in weights that connect to the next layer of neurons in a densely connected layer in any feed forward neural network. The activity-weight (A-W) duality allows us to map variations in inputs (data) to variations of the corresponding dual weights. By using this mapping, we show that the generalization loss can be decomposed into a sum of contributions from different eigen-directions of the Hessian matrix of the loss function at the solution in weight space. The contribution from a given eigen-direction is the product of two geometric factors (determinants): the sharpness of the loss landscape and the standard deviation of the dual weights, which is found to scale with the weight norm of the solution. Our results provide an unified framework, which we used to reveal how different regularization schemes (weight decay, stochastic gradient descent with different batch sizes and learning rates, dropout), training data size, and labeling noise affect generalization performance by controlling either one or both of these two geometric determinants for generalization. These insights can be used to guide development of algorithms for finding more generalizable solutions in overparametrized neural networks.

翻译：机器学习的基本问题之一是泛化能力。在具有大量权重（参数）的神经网络模型中，可能存在许多能够同等拟合训练数据的解。关键问题在于，哪一个解能够描述不在训练集中的测试数据。本文报告了一项重要发现：在任何前馈神经网络的全连接层中，给定层神经元活动的变化与连接至下一层神经元的权重变化之间存在精确的对偶性（等价关系）。活动-权重（A-W）对偶性使我们能够将输入（数据）的变化映射为对应对偶权重的变化。利用这一映射，我们证明了泛化损失可以分解为损失函数在权重空间解处的Hessian矩阵不同特征方向上的贡献之和。某一特定特征方向的贡献是两个几何因子（决定因素）的乘积：损失景观的尖锐度与对偶权重的标准差，后者被发现与该解的权重范数成比例。我们的结果提供了一个统一框架，并利用该框架揭示了不同正则化方案（权重衰减、不同批量大小和学习率的随机梯度下降、dropout）、训练数据规模以及标签噪声如何通过控制泛化能力的这两个几何决定因素之一或两者来影响泛化性能。这些见解可用于指导开发算法，以在过参数化神经网络中寻找更具泛化能力的解。