Existing work has shown that cross-validation (CV) can be used to provide an asymptotic confidence interval for the test error of a stable machine learning algorithm, and existing stability results for many popular algorithms can be applied to derive positive instances where such confidence intervals will be valid. However, in the common setting where CV is used to compare two algorithms, it becomes necessary to consider a notion of relative stability which cannot easily be derived from existing stability results, even for simple algorithms. To better understand relative stability and when CV provides valid confidence intervals for the test error difference of two algorithms, we study the soft-thresholded least squares algorithm, a close cousin of the Lasso. We prove that while stability holds when assessing the individual test error of this algorithm, relative stability fails to hold when comparing the test error of two such algorithms, even in a sparse low-dimensional linear model setting. Additionally, we empirically confirm the invalidity of CV confidence intervals for the test error difference when either soft-thresholding or the Lasso is used. In short, caution is needed when quantifying the uncertainty of CV estimates of the performance difference of two machine learning algorithms, even when both algorithms are individually stable.

翻译：现有研究表明，交叉验证（CV）可用于为稳定机器学习算法的测试误差提供渐近置信区间，且许多流行算法的现有稳定性结果可用于推导此类置信区间有效的积极实例。然而，在CV被用于比较两种算法的常见场景中，必须考虑相对稳定性的概念，而这一概念难以从现有稳定性结果中推导得出，即使对于简单算法亦是如此。为更好理解相对稳定性以及CV何时能为两种算法的测试误差差异提供有效置信区间，我们研究了软阈值最小二乘算法（Lasso算法的近亲）。我们证明，虽然评估该算法个体测试误差时稳定性成立，但在比较两种此类算法的测试误差时，即使是在稀疏低维线性模型设定中，相对稳定性也无法成立。此外，我们通过实证研究证实，当使用软阈值法或Lasso算法时，针对测试误差差异的CV置信区间均无效。简言之，在量化两种机器学习算法性能差异的CV估计不确定性时需保持谨慎，即使两种算法各自均具有稳定性。