Receiver operating characteristic (ROC) analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under ROC curve (AUC) with standard cross-validation methods suffers from a large bias. The leave-pair-out (LPO) cross-validation has been shown to correct this bias. However, while LPO produces an almost unbiased estimate of AUC, it does not provide a ranking of the data needed for plotting and analyzing the ROC curve. In this study, we propose a new method called tournament leave-pair-out (TLPO) cross-validation. This method extends LPO by creating a tournament from pair comparisons to produce a ranking for the data. TLPO preserves the advantage of LPO for estimating AUC, while it also allows performing ROC analyses. We have shown using both synthetic and real world data that TLPO is as reliable as LPO for AUC estimation, and confirmed the bias in leave-one-out cross-validation on low-dimensional data. As a case study on ROC analysis, we also evaluate how reliably sensitivity and specificity can be estimated from TLPO ROC curves.

翻译：受试者工作特征（ROC）分析被广泛用于评估诊断系统。近期研究表明，采用标准交叉验证方法估计ROC曲线下面积（AUC）存在较大偏差。留配对（LPO）交叉验证已被证明能够纠正这一偏差。然而，尽管LPO可产生近乎无偏的AUC估计值，但其无法提供绘制和分析ROC曲线所需的数据排序。本研究提出一种名为锦标赛式留配对（TLPO）交叉验证的新方法。该方法通过将配对比对结果构建为锦标赛形式对数据进行排序，从而扩展了LPO方法。TLPO既保留了LPO估计AUC的优势，又能够执行ROC分析。通过合成数据与真实数据验证表明，TLPO在AUC估计方面与LPO具有同等可靠性，同时证实了低维数据中留一交叉验证存在的偏差。作为ROC分析的案例研究，我们还评估了从TLPO ROC曲线中估计敏感度与特异度的可靠性。