Two key tasks in high-dimensional regularized regression are tuning the regularization strength for accurate predictions and estimating the out-of-sample risk. It is known that the standard approach -- $k$-fold cross-validation -- is inconsistent in modern high-dimensional settings. While leave-one-out and generalized cross-validation remain consistent in some high-dimensional cases, they become inconsistent when samples are dependent or contain heavy-tailed covariates. As a first step towards modeling structured sample dependence and heavy tails, we use right-rotationally invariant covariate distributions -- a crucial concept from compressed sensing. In the proportional asymptotics regime where the number of features and samples grow comparably, which is known to better reflect the empirical behavior in moderately sized datasets, we introduce a new framework, ROTI-GCV, for reliably performing cross-validation under these challenging conditions. Along the way, we propose new estimators for the signal-to-noise ratio and noise variance. We conduct experiments that demonstrate the accuracy of our approach in a variety of synthetic and semi-synthetic settings.

翻译：高维正则化回归中的两个关键任务是调整正则化强度以获得准确预测，以及估计样本外风险。众所周知，标准方法——$k$折交叉验证——在现代高维场景下是不一致的。尽管留一法和广义交叉验证在某些高维情况下仍能保持一致性，但当样本存在依赖性或包含重尾协变量时，它们也会变得不一致。作为建模结构化样本依赖性和重尾分布的第一步，我们采用右旋转不变协变量分布——这是压缩感知中的一个核心概念。在特征数与样本数可比增长的渐近比例框架下（该框架被认为能更好地反映中等规模数据集的经验行为），我们提出了一个新框架ROTI-GCV，用于在这些挑战性条件下可靠地进行交叉验证。在此过程中，我们提出了信噪比与噪声方差的新估计量。我们通过实验验证了该方法在各种合成与半合成场景中的准确性。