This paper studies generalization capabilities of neural networks (NNs) using new and improved PyTorch library Loss Landscape Analysis (LLA). LLA facilitates visualization and analysis of loss landscapes along with the properties of NN Hessian. Different approaches to NN loss landscape plotting are discussed with particular focus on normalization techniques showing that conventional methods cannot always ensure correct visualization when batch normalization layers are present in NN architecture. The use of Hessian axes is shown to be able to mitigate this effect, and methods for choosing Hessian axes are proposed. In addition, spectra of Hessian eigendecomposition are studied and it is shown that typical spectra exist for a wide range of NNs. This allows to propose quantitative criteria for Hessian analysis that can be applied to evaluate NN performance and assess its generalization capabilities. Generalization experiments are conducted using ImageNet-1K pre-trained models along with several models trained as part of this study. The experiment include training models on one dataset and testing on another one to maximize experiment similarity to model performance in the Wild. It is shown that when datasets change, the changes in criteria correlate with the changes in accuracy, making the proposed criteria a computationally efficient estimate of generalization ability, which is especially useful for extremely large datasets.

翻译：本研究利用全新改进的PyTorch库Loss Landscape Analysis（LLA）探究神经网络的泛化能力。该库支持损失景观的可视化分析及神经网络Hessian矩阵的特性研究。论文系统讨论了神经网络损失景观的绘制方法，重点聚焦于归一化技术，并指出当网络架构包含批归一化层时，传统方法无法保证可视化的准确性。研究表明采用Hessian轴可有效缓解此问题，同时提出了Hessian轴的选取方法。此外，通过对Hessian特征分解谱的分析，发现广泛类型的神经网络存在典型谱特征。基于此，我们提出了可量化评估神经网络性能及其泛化能力的Hessian分析准则。泛化实验采用ImageNet-1K预训练模型及本研究训练的部分模型，通过在特定数据集训练并在另一数据集测试的方式，最大程度模拟模型在真实场景中的性能表现。实验表明当数据集变更时，所提准则的变化与准确率变化具有相关性，这使其成为计算高效的泛化能力评估指标，对于超大规模数据集的应用场景尤为适用。