To evaluate a classification algorithm, it is common practice to plot the ROC curve using test data. However, the inherent randomness in the test data can undermine our confidence in the conclusions drawn from the ROC curve, necessitating uncertainty quantification. In this article, we propose an algorithm to construct confidence bands for the ROC curve, quantifying the uncertainty of classification on the test data in terms of sensitivity and specificity. The algorithm is based on a procedure called conformal prediction, which constructs individualized confidence intervals for the test set and the confidence bands for the ROC curve can be obtained by combining the individualized intervals together. Furthermore, we address both scenarios where the test data are either iid or non-iid relative to the observed data set and propose distinct algorithms for each case with valid coverage probability. The proposed method is validated through both theoretical results and numerical experiments.

翻译：为评估分类算法，通常做法是使用测试数据绘制ROC曲线。然而，测试数据固有的随机性可能削弱我们从ROC曲线中得出结论的可靠性，因此需要进行不确定性量化。本文提出了一种构建ROC曲线置信带的算法，从敏感性和特异性角度量化测试数据分类的不确定性。该算法基于一种称为共形预测的方法，为测试集构造个性化置信区间，并通过组合这些个性化区间得到ROC曲线的置信带。此外，我们分别考虑了测试数据相对于观测数据集独立同分布与非独立同分布两种情形，针对每种情形提出了具有有效覆盖概率的独立算法。通过理论结果与数值实验验证了所提方法的有效性。