Online nonparametric estimators are gaining popularity due to their efficient computation and competitive generalization abilities. An important example includes variants of stochastic gradient descent. These algorithms often take one sample point at a time and instantly update the parameter estimate of interest. In this work we consider model selection and hyperparameter tuning for such online algorithms. We propose a weighted rolling-validation procedure, an online variant of leave-one-out cross-validation, that costs minimal extra computation for many typical stochastic gradient descent estimators. Similar to batch cross-validation, it can boost base estimators to achieve a better, adaptive convergence rate. Our theoretical analysis is straightforward, relying mainly on some general statistical stability assumptions. The simulation study underscores the significance of diverging weights in rolling validation in practice and demonstrates its sensitivity even when there is only a slim difference between candidate estimators.

翻译：在线非参数估计因其高效计算和良好的泛化能力而日益受到关注，其中随机梯度下降的变体是重要示例。这类算法通常每次只取一个样本点，并即时更新目标参数估计值。本文针对此类在线算法，研究模型选择与超参数调优问题。我们提出加权滚动验证方法——一种留一交叉验证的在线变体，该方法在诸多典型随机梯度下降估计器中仅需极少的额外计算代价。与批处理交叉验证类似，它能提升基估计器性能，实现更优的自适应收敛速度。我们的理论分析简明直观，主要依赖若干通用统计稳定性假设。仿真研究凸显了滚动验证中权重差异化的实践意义，并证明即便候选估计器之间差异微小，该方法仍能保持敏感性。