A user-focused verification approach for evaluating probability forecasts of binary outcomes (also known as probabilistic classifiers) is demonstrated that is (i) based on proper scoring rules, (ii) focuses on user decision thresholds, and (iii) provides actionable insights. It is argued that when categorical performance diagrams and the critical success index are used to evaluate overall predictive performance, rather than the discrimination ability of probabilistic forecasts, they may produce misleading results. Instead, Murphy diagrams are shown to provide better understanding of overall predictive performance as a function of user probabilistic decision threshold. It is illustrated how to select a proper scoring rule, based on the relative importance of different user decision thresholds, and how this choice impacts scores of overall predictive performance and supporting measures of discrimination and calibration. These approaches and ideas are demonstrated using several probabilistic thunderstorm forecast systems as well as synthetic forecast data. Furthermore, a fair method for comparing the performance of probabilistic and categorical forecasts is illustrated using the FIxed Risk Multicategorical (FIRM) score, which is a proper scoring rule directly connected to values on the Murphy diagram. While the methods are illustrated using thunderstorm forecasts, they are applicable for evaluating probabilistic forecasts for any situation with binary outcomes.

翻译：本文展示了一种以用户为中心的二元结果概率预测（亦称概率分类器）验证方法，该方法基于：（i）恰当评分规则；（ii）聚焦用户决策阈值；（iii）提供可操作见解。论证表明，当使用分类性能图和关键成功指数评估整体预测性能（而非概率预测的区分能力）时，可能产生误导性结果。相反，墨菲图能够根据用户概率决策阈值更清晰地揭示整体预测性能。本文阐述了如何根据不同用户决策阈值的相对重要性选择恰当的评分规则，以及该选择如何影响整体预测性能评分及区分度与校准度等辅助度量指标。通过多个雷暴概率预报系统及合成预报数据对上述方法与思路进行了验证。此外，本文还利用固定风险多分类（FIRM）评分——一种直接关联墨菲图数值的恰当评分规则——展示了公平比较概率预测与分类预测性能的方法。尽管以雷暴预报为例进行方法演示，但该框架适用于任何二元结果场景下的概率预测评估。