Recently, deep neural networks have been found to nearly interpolate training data but still generalize well in various applications. To help understand such a phenomenon, it has been of interest to analyze the ridge estimator and its interpolation limit in high-dimensional regression models. For this motivation, we study the ridge estimator in a rotationally sparse setting of high-dimensional linear regression, where the signal of a response is aligned with a small number, $d$, of covariates with large or spiked variances, compared with the remaining covariates with small or tail variances, \textit{after} an orthogonal transformation of the covariate vector. We establish high-probability upper and lower bounds on the out-sample and in-sample prediction errors in two distinct regimes depending on the ratio of the effective rank of tail variances over the sample size $n$. The separation of the two regimes enables us to exploit relevant concentration inequalities and derive concrete error bounds without making any oracle assumption or independent components assumption on covariate vectors. Moreover, we derive sufficient and necessary conditions which indicate that the prediction errors of ridge estimation can be of the order $O(\frac{d}{n})$ if and only if the gap between the spiked and tail variances are sufficiently large. We also compare the orders of optimal out-sample and in-sample prediction errors and find that, remarkably, the optimal out-sample prediction error may be significantly smaller than the optimal in-sample one. Finally, we present numerical experiments which empirically confirm our theoretical findings.

翻译：最近，深度神经网络在多类应用中被发现几乎能够精确插值训练数据，同时仍具有良好的泛化能力。为理解这一现象，分析高维回归模型中的岭估计量及其插值极限已成为研究重点。基于此动机，我们研究了高维线性回归旋转稀疏设定下的岭估计量：在对协变量向量进行正交变换后，响应信号的强度集中于少数$d$个具有大方差或尖峰方差的协变量上，而其余协变量则具有小方差或尾部方差。我们根据尾部方差有效秩与样本量$n$的比值，在两种不同机制下建立了样本外与样本内预测误差的高概率上下界。两种机制的分离使我们能够利用相关集中不等式，在无需对协变量向量作预言机假设或独立成分假设的情况下推导出具体误差界。此外，我们推导了充分必要条件，表明尖峰方差与尾部方差的差距足够大时，岭估计的预测误差可达$O(\frac{d}{n})$阶。同时，通过比较最优样本外与样本内预测误差的阶，发现最优样本外预测误差可能显著小于最优样本内预测误差。最后，我们通过数值实验验证了理论结果。