Deep neural networks (DNNs) have made great strides in pushing the state-of-the-art in several challenging domains. Recent studies reveal that they are prone to making overconfident predictions. This greatly reduces the overall trust in model predictions, especially in safety-critical applications. Early work in improving model calibration employs post-processing techniques which rely on limited parameters and require a hold-out set. Some recent train-time calibration methods, which involve all model parameters, can outperform the postprocessing methods. To this end, we propose a new train-time calibration method, which features a simple, plug-and-play auxiliary loss known as multi-class alignment of predictive mean confidence and predictive certainty (MACC). It is based on the observation that a model miscalibration is directly related to its predictive certainty, so a higher gap between the mean confidence and certainty amounts to a poor calibration both for in-distribution and out-of-distribution predictions. Armed with this insight, our proposed loss explicitly encourages a confident (or underconfident) model to also provide a low (or high) spread in the presoftmax distribution. Extensive experiments on ten challenging datasets, covering in-domain, out-domain, non-visual recognition and medical image classification scenarios, show that our method achieves state-of-the-art calibration performance for both in-domain and out-domain predictions. Our code and models will be publicly released.

翻译：深度神经网络（DNNs）在多个挑战性领域取得了显著进展，推动了前沿技术的发展。近期研究表明，这些网络容易做出过度自信的预测，这极大地降低了模型预测的整体可信度，尤其是在安全关键型应用中。早期改善模型校准的工作采用后处理技术，此类技术依赖有限参数并需要独立验证集。而近年来一些涉及全部模型参数的训练时校准方法，其性能可超越后处理方法。为此，我们提出一种新的训练时校准方法，其核心是一个简单且即插即用的辅助损失函数——预测均值置信度与预测确定性的多类别对齐（MACC）。该方法基于以下观察：模型校准误差与其预测确定性直接相关，因此均值置信度与确定性之间的差距越大，无论是在分布内还是分布外预测中，校准效果越差。基于这一发现，我们提出的损失函数明确鼓励自信（或欠自信）模型在softmax前的分布中呈现低（或高）离散度。在涵盖域内、域外、非视觉识别及医学图像分类等场景的十个挑战性数据集上的大量实验表明，我们的方法在域内和域外预测中均达到了最先进的校准性能。我们的代码和模型将公开发布。