We study generalization properties of random features (RF) regression in high dimensions optimized by stochastic gradient descent (SGD). In this regime, we derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting, and observe the double descent phenomenon both theoretically and empirically. Our analysis shows how to cope with multiple randomness sources of initialization, label noise, and data sampling (as well as stochastic gradients) with no closed-form solution, and also goes beyond the commonly-used Gaussian/spherical data assumption. Our theoretical results demonstrate that, with SGD training, RF regression still generalizes well for interpolation learning, and is able to characterize the double descent behavior by the unimodality of variance and monotonic decrease of bias. Besides, we also prove that the constant step-size SGD setting incurs no loss in convergence rate when compared to the exact minimal-norm interpolator, as a theoretical justification of using SGD in practice.

翻译：我们的研究是随机特征(RF)回归的概括性特征,这种回归在通过随机梯度梯度下最优化的高维范围内。在这个制度下,我们从恒定和适应性级的SGD设置中得出精确的不被动的RF回归误差界限,在理论上和经验上观察双重下降现象。我们的分析表明如何应对启动、标签噪声和数据抽样(以及随机梯度梯度)的多种随机来源,而没有封闭式解决方案,而且超出了常用的高频/球状数据假设。我们的理论结果表明,在SGD培训中,RF回归仍然对内插学习进行精准的概括化,并且能够通过差异的不单一形式和偏见的单向下降来描述双向下降的双重下降行为。此外,我们还证明,与精确的最小北向内插器相比,不变的SGD设置不会造成趋同率的损失,作为在实践中使用SGD的理论依据。