Advanced classification algorithms are being increasingly used in safety-critical applications like health-care, engineering, etc. In such applications, miss-classifications made by ML algorithms can result in substantial financial or health-related losses. To better anticipate and prepare for such losses, the algorithm user seeks an estimate for the probability that the algorithm miss-classifies a sample. We refer to this task as the risk-assessment. For a variety of models and datasets, we numerically analyze the performance of different methods in solving the risk-assessment problem. We consider two solution strategies: a) calibration techniques that calibrate the output probabilities of classification models to provide accurate probability outputs; and b) a novel approach based upon the prediction interval generation technique of conformal prediction. Our conformal prediction based approach is model and data-distribution agnostic, simple to implement, and provides reasonable results for a variety of use-cases. We compare the different methods on a broad variety of models and datasets.

翻译：高级分类算法正日益广泛应用于医疗保健、工程等安全关键领域。在此类应用中，机器学习算法的误分类可能导致重大的经济损失或健康损害。为更好地预测和应对此类损失，算法使用者需要估计算法对样本进行误分类的概率。我们将此任务称为风险评估。针对多种模型和数据集，我们数值分析了不同方法在解决风险评估问题时的性能表现。我们考虑两种解决策略：a) 通过校准技术调整分类模型的输出概率以提供精确的概率输出；b) 基于保形预测的预测区间生成技术提出创新方法。我们提出的保形预测方法具有模型无关性与数据分布无关性，实现简单，且在多种应用场景中均能提供合理结果。我们在广泛的模型与数据集上对各类方法进行了比较分析。