Deep neural networks are increasingly utilized in various machine learning tasks. However, as these models grow in complexity, they often face calibration issues, despite enhanced prediction accuracy. Many studies have endeavored to improve calibration performance through the use of specific loss functions, data preprocessing and training frameworks. Yet, investigations into calibration properties have been somewhat overlooked. Our study leverages the Neural Architecture Search (NAS) search space, offering an exhaustive model architecture space for thorough calibration properties exploration. We specifically create a model calibration dataset. This dataset evaluates 90 bin-based and 12 additional calibration measurements across 117,702 unique neural networks within the widely employed NATS-Bench search space. Our analysis aims to answer several longstanding questions in the field, using our proposed dataset: (i) Can model calibration be generalized across different datasets? (ii) Can robustness be used as a calibration measurement? (iii) How reliable are calibration metrics? (iv) Does a post-hoc calibration method affect all models uniformly? (v) How does calibration interact with accuracy? (vi) What is the impact of bin size on calibration measurement? (vii) Which architectural designs are beneficial for calibration? Additionally, our study bridges an existing gap by exploring calibration within NAS. By providing this dataset, we enable further research into NAS calibration. As far as we are aware, our research represents the first large-scale investigation into calibration properties and the premier study of calibration issues within NAS. The project page can be found at https://www.taolinwei.com/calibration-study

翻译：深度神经网络正日益广泛应用于各类机器学习任务中。然而，随着这些模型复杂度的提升，尽管预测精度有所提高，但它们常常面临校准问题。许多研究致力于通过使用特定损失函数、数据预处理和训练框架来改善校准性能。然而，对校准特性的研究在某种程度上被忽视了。我们的研究利用神经架构搜索（NAS）搜索空间，提供了一个详尽的模型架构空间，以进行全面的校准特性探索。我们专门创建了一个模型校准数据集。该数据集在广泛使用的NATS-Bench搜索空间中，评估了117,702个独特神经网络上的90项基于分箱的校准度量及12项额外校准度量。我们的分析旨在利用所提出的数据集回答该领域几个长期存在的问题：（i）模型校准能否跨不同数据集泛化？（ii）鲁棒性能否用作校准度量？（iii）校准指标的可信度如何？（iv）事后校准方法是否对所有模型产生一致影响？（v）校准与准确度如何交互？（vi）分箱大小对校准度量有何影响？（vii）哪些架构设计有利于校准？此外，我们的研究通过探索NAS中的校准，填补了现有空白。通过提供该数据集，我们得以推动对NAS校准的进一步研究。据我们所知，我们的研究代表了首次针对校准特性的大规模调查，也是NAS中校准问题的开创性研究。项目页面可在https://www.taolinwei.com/calibration-study查阅。