Cross-validation is a widely used technique for evaluating the performance of prediction models. It helps avoid the optimism bias in error estimates, which can be significant for models built using complex statistical learning algorithms. However, since the cross-validation estimate is a random value dependent on observed data, it is essential to accurately quantify the uncertainty associated with the estimate. This is especially important when comparing the performance of two models using cross-validation, as one must determine whether differences in error estimates are a result of chance fluctuations. Although various methods have been developed for making inferences on cross-validation estimates, they often have many limitations, such as stringent model assumptions This paper proposes a fast bootstrap method that quickly estimates the standard error of the cross-validation estimate and produces valid confidence intervals for a population parameter measuring average model performance. Our method overcomes the computational challenge inherent in bootstrapping the cross-validation estimate by estimating the variance component within a random effects model. It is just as flexible as the cross-validation procedure itself. To showcase the effectiveness of our approach, we employ comprehensive simulations and real data analysis across three diverse applications.

翻译：交叉验证是评估预测模型性能的广泛使用技术，能够有效避免误差估计中的乐观偏差——这种偏差对基于复杂统计学习算法构建的模型尤为显著。然而，由于交叉验证估计值依赖于观测数据且具有随机性，准确量化该估计值的不确定性至关重要。当通过交叉验证比较两个模型性能时，这一点尤为关键——研究者需要判断误差估计差异是否源于随机波动。尽管已有多种方法可用于推断交叉验证估计值，但这些方法往往存在诸多局限，例如严格的模型假设。本文提出一种快速自助法，可高效估计交叉验证估计值的标准误差，并为衡量平均模型性能的总体参数生成有效置信区间。本方法通过估计随机效应模型中的方差分量，克服了自助法交叉验证估计固有的计算难题，其灵活性与交叉验证流程本身相当。为展示本方法的有效性，我们通过三项跨领域应用的全面模拟与真实数据分析进行了验证。