Evaluating the performance of classifiers is critical in machine learning, particularly in high-stakes applications where the reliability of predictions can significantly impact decision-making. Traditional performance measures, such as accuracy and F-score, often fail to account for the uncertainty inherent in classifier predictions, leading to potentially misleading assessments. This paper introduces the Certainty Ratio ($C_\rho$), a novel metric designed to quantify the contribution of confident (certain) versus uncertain predictions to any classification performance measure. By integrating the Probabilistic Confusion Matrix ($CM^\star$) and decomposing predictions into certainty and uncertainty components, $C_\rho$ provides a more comprehensive evaluation of classifier reliability. Experimental results across 21 datasets and multiple classifiers, including Decision Trees, Naive-Bayes, 3-Nearest Neighbors, and Random Forests, demonstrate that $C_\rho$ reveals critical insights that conventional metrics often overlook. These findings emphasize the importance of incorporating probabilistic information into classifier evaluation, offering a robust tool for researchers and practitioners seeking to improve model trustworthiness in complex environments.

翻译：评估分类器的性能在机器学习中至关重要，尤其是在高风险应用中，预测的可靠性可能显著影响决策。传统的性能指标，如准确率和F分数，往往未能考虑分类器预测中固有的不确定性，可能导致具有误导性的评估。本文提出确定性比率（$C_\rho$），这是一种旨在量化自信（确定）预测与不确定预测对任何分类性能指标贡献的新指标。通过整合概率混淆矩阵（$CM^\star$）并将预测分解为确定性和不确定性分量，$C_\rho$ 提供了对分类器可靠性更全面的评估。在21个数据集和多种分类器（包括决策树、朴素贝叶斯、3-最近邻和随机森林）上的实验结果表明，$C_\rho$ 揭示了传统指标常常忽略的关键洞察。这些发现强调了将概率信息纳入分类器评估的重要性，为研究者和从业者在复杂环境中寻求提升模型可信度提供了一个稳健的工具。