Reliable confidence estimation is a challenging yet fundamental requirement in many risk-sensitive applications. However, modern deep neural networks are often overconfident for their incorrect predictions, i.e., misclassified samples from known classes, and out-of-distribution (OOD) samples from unknown classes. In recent years, many confidence calibration and OOD detection methods have been developed. In this paper, we find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We investigate this problem and reveal that popular calibration and OOD detection methods often lead to worse confidence separation between correctly classified and misclassified examples, making it difficult to decide whether to trust a prediction or not. Finally, we propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance under various settings including balanced, long-tailed, and covariate-shift classification scenarios. Our study not only provides a strong baseline for reliable confidence estimation but also acts as a bridge between understanding calibration, OOD detection, and failure prediction. The code is available at \url{https://github.com/Impression2805/FMFP}.

翻译：可靠的置信度估计是许多风险敏感应用中一项具有挑战性但至关重要的基本要求。然而，现代深度神经网络往往对其错误预测过于自信，即来自已知类别的误分类样本和来自未知类别的分布外样本。近年来，许多置信度校准和分布外检测方法已被开发出来。在本文中，我们发现了一个普遍存在但实际被忽视的现象：大多数置信度估计方法对检测误分类错误是有害的。我们研究了这个问题，并揭示了流行的校准和分布外检测方法往往导致正确分类与误分类样本之间的置信度分离更差，使得难以决定是否信任一个预测。最后，我们提出通过寻找平坦最小值来扩大置信度差距，这在包括平衡、长尾和协变量偏移分类场景在内的各种设置下取得了最先进的失败预测性能。我们的研究不仅为可靠的置信度估计提供了强基线，还充当了理解校准、分布外检测和失败预测之间的桥梁。代码可在 \url{https://github.com/Impression2805/FMFP} 获取。