Stochastic gradient descent (SGD) or stochastic approximation has been widely used in model training and stochastic optimization. While there is a huge literature on analyzing its convergence, inference on the obtained solutions from SGD has only been recently studied, yet is important due to the growing need for uncertainty quantification. We investigate two computationally cheap resampling-based methods to construct confidence intervals for SGD solutions. One uses multiple, but few, SGDs in parallel via resampling with replacement from the data, and another operates this in an online fashion. Our methods can be regarded as enhancements of established bootstrap schemes to substantially reduce the computation effort in terms of resampling requirements, while at the same time bypassing the intricate mixing conditions in existing batching methods. We achieve these via a recent so-called cheap bootstrap idea and Berry-Esseen-type bound for SGD.

翻译：随机梯度下降（SGD）或随机逼近方法已广泛应用于模型训练与随机优化领域。尽管其收敛性分析已有大量文献研究，但受不确定性量化日益增长的需求驱动，SGD所得解的统计推断问题直至近期才引起学界关注。本文提出两种计算成本低廉的基于重采样方法，用于构建SGD解的置信区间：其一通过数据有放回重采样并行运行少量（但多于单个）SGD，其二以在线方式实现该过程。我们的方法可视为对传统自助法方案的改进，通过大幅降低重采样所需的计算量，同时规避现有批处理方法中复杂的混合条件约束。这些改进得益于近期提出的所谓廉价自助法思路，以及面向SGD的Berry-Esseen型界。