Online nonparametric estimators are gaining popularity due to their efficient computation and competitive generalization abilities. An important example includes variants of stochastic gradient descent. These algorithms often take one sample point at a time and instantly update the parameter estimate of interest. In this work we consider model selection and hyperparameter tuning for such online algorithms. We propose a weighted rolling-validation procedure, an online variant of leave-one-out cross-validation, that costs minimal extra computation for many typical stochastic gradient descent estimators. Similar to batch cross-validation, it can boost base estimators to achieve a better, adaptive convergence rate. Our theoretical analysis is straightforward, relying mainly on some general statistical stability assumptions. The simulation study underscores the significance of diverging weights in rolling validation in practice and demonstrates its sensitivity even when there is only a slim difference between candidate estimators.

翻译：在线非参数估计因其高效的计算和竞争性的泛化能力而日益受到关注，其中随机梯度下降的变体是一个重要例子。这些算法通常每次仅处理一个样本点，并即时更新所关注的参数估计。在本工作中，我们考虑针对此类在线算法的模型选择与超参数调优。我们提出了一种加权滚动验证过程，这是留一交叉验证的在线变体，对于许多典型的随机梯度下降估计器而言，其额外计算成本极低。与批量交叉验证类似，该方法可提升基础估计器的性能，使其达到更优的自适应收敛速度。我们的理论分析较为简洁，主要依赖于一些通用的统计稳定性假设。仿真研究强调了滚动验证中发散权重在实践中的重要性，并证明即使在候选估计器差异微小的情况下，该方法仍具有敏感性。