Sharpness of minima is a promising quantity that can correlate with generalization in deep networks and, when optimized during training, can improve generalization. However, standard sharpness is not invariant under reparametrizations of neural networks, and, to fix this, reparametrization-invariant sharpness definitions have been proposed, most prominently adaptive sharpness (Kwon et al., 2021). But does it really capture generalization in modern practical settings? We comprehensively explore this question in a detailed study of various definitions of adaptive sharpness in settings ranging from training from scratch on ImageNet and CIFAR-10 to fine-tuning CLIP on ImageNet and BERT on MNLI. We focus mostly on transformers for which little is known in terms of sharpness despite their widespread usage. Overall, we observe that sharpness does not correlate well with generalization but rather with some training parameters like the learning rate that can be positively or negatively correlated with generalization depending on the setup. Interestingly, in multiple cases, we observe a consistent negative correlation of sharpness with out-of-distribution error implying that sharper minima can generalize better. Finally, we illustrate on a simple model that the right sharpness measure is highly data-dependent, and that we do not understand well this aspect for realistic data distributions. The code of our experiments is available at https://github.com/tml-epfl/sharpness-vs-generalization.

翻译：摘要：最小值的锐度是一个有前景的量，它能与深度网络中的泛化性相关联，且在训练过程中进行优化时可提升泛化能力。然而，标准锐度在神经网络重参数化下不具备不变性，为此研究者提出了重参数化不变的锐度定义，其中最具代表性的是自适应锐度（Kwon等人，2021年）。但在现代实际场景中，它是否真正能反映泛化性？我们通过详细研究多种自适应锐度定义，全面探讨了这一问题，涵盖从在ImageNet和CIFAR-10上从头训练，到在ImageNet上微调CLIP和在MNLI上微调BERT等场景。我们主要聚焦于Transformer模型，尽管其应用广泛，但其锐度特性鲜有研究。总体而言，我们观察到锐度与泛化性并非高度相关，而是与某些训练参数（如学习率）存在关联，而学习率与泛化性的相关性根据设置可能为正或负。有趣的是，在多种情况下，我们发现锐度与分布外误差存在一致的负相关，这意味着更尖锐的最小值可能具有更好的泛化性。最后，我们通过一个简单模型说明，正确的锐度度量高度依赖于数据，而我们对真实数据分布下这一方面的理解仍不充分。实验代码已发布于https://github.com/tml-epfl/sharpness-vs-generalization。