Classification models play a central role in data-driven decision-making applications such as medical diagnosis, recommendation systems, and risk assessment. Traditional performance metrics, such as accuracy and AUC, focus on overall error rates but fail to account for the confidence of incorrect predictions, i.e., the risk of confident misjudgments. This limitation is particularly consequential in safety-critical and cost-sensitive settings, where overconfident errors can lead to severe outcomes. To address this issue, we propose the Fragility Index (FI), a novel performance metric that evaluates classifiers from a risk-averse perspective by capturing the tail risk of confident misjudgments. We formulate FI within a robust satisficing (RS) framework to ensure robustness under distributional uncertainty. Building on this, we develop a tractable training framework that directly targets FI via a surrogate loss, and show that models trained under this framework admit provable bounds on FI. We further derive exact reformulations for a broad class of loss functions, including cross-entropy, hinge-type, and Lipschitz losses, and extend the approach to deep neural networks. Empirical results on real-world medical diagnosis tasks demonstrate that FI complements existing metrics by revealing error tail risk and improving decision quality. FI-based models achieve competitive accuracy and AUC while consistently reducing confident misjudgments and associated operational costs, offering a practical tool for improving robustness and reliability in risk-critical applications.

翻译：分类模型在数据驱动型决策应用中扮演核心角色，如医疗诊断、推荐系统和风险评估。传统性能指标（如准确率和AUC）关注总体错误率，但未能考虑错误预测的置信度，即自信误判的风险。这一局限在安全关键型及成本敏感型场景中尤为严重——过度自信的错误可能导致严重后果。为解决该问题，我们提出脆弱性指数（FI），这是一种从风险规避视角评估分类器的新指标，通过捕获自信误判的尾部风险来度量性能。我们基于鲁棒满足性（RS）框架构建FI，以确保在分布不确定性下的鲁棒性。在此基础上，我们开发了一个可直接通过替代损失优化FI的可训练框架，并证明在该框架下训练的模型可获得FI的保证边界。我们进一步导出了包括交叉熵、铰链型和Lipschitz损失在内的大类损失函数的精确重构形式，并将该方法扩展至深度神经网络。在真实医疗诊断任务上的实证结果表明，FI通过揭示错误尾部风险并提升决策质量，与现有指标形成互补。基于FI的模型在保持竞争力准确率和AUC的同时，持续减少自信误判及相关操作成本，为风险关键型应用的鲁棒性与可靠性提升提供了实用工具。