We study the unconstrained minimization of a smooth and strongly convex population loss function under a stochastic oracle that introduces both additive and multiplicative noise; this is a canonical and widely-studied setting that arises across operations research, signal processing, and machine learning. We begin by showing that standard approaches such as sample average approximation and robust (or averaged) stochastic approximation can lead to suboptimal -- and in some cases arbitrarily poor -- performance with realistic finite sample sizes. In contrast, we demonstrate that a carefully designed variance reduction strategy, which we term VISOR for short, can significantly outperform these approaches while using the same sample size. Our upper bounds are complemented by finite-sample, information-theoretic local minimax lower bounds, which highlight fundamental, instance-dependent factors that govern the performance of any estimator. Taken together, these results demonstrate that an accelerated variant of VISOR is instance-optimal, achieving the best possible sample complexity up to logarithmic factors while also attaining optimal oracle complexity. We apply our theory to generalized linear models and improve upon classical results. In particular, we obtain the best-known non-asymptotic, instance-dependent generalization error bounds for stochastic methods, even in linear regression.

翻译：我们研究在同时引入加性和乘性噪声的随机预言机下，对光滑且强凸的总体损失函数进行无约束最小化问题；这是一个跨运筹学、信号处理和机器学习领域的经典且广泛研究的设定。我们首先证明，标准方法（如样本平均近似和鲁棒（或平均）随机逼近）在现实有限样本量下可能导致次优——甚至在某些情况下任意差——的性能。相比之下，我们展示了一种精心设计的方差缩减策略（我们简称为VISOR）能够在相同样本量下显著优于这些方法。我们的上界通过有限样本、信息论局部极小极大下界进行补充，这些下界揭示了支配任何估计器性能的基础性、实例依赖因素。综合这些结果，我们证明VISOR的加速变体是实例最优的，在对数因子范围内实现了最佳样本复杂度，同时达到最优的预言机复杂度。我们将理论应用于广义线性模型，并改进了经典结果。特别地，我们为随机方法（甚至在线性回归中）获得了目前最佳的非渐近性、实例依赖的泛化误差界。