Empirical risk minimization on massive datasets naturally exhibits a nested double finite-sum structure, where $N=nm$ total samples are logically or physically partitioned into $n$ blocks of size $m$ (e.g., in pooled data silos, out-of-core learning, or deliberate stratification). While variance-reduced methods achieve optimal oracle complexities for nonconvex objectives, they suffer from severe scaling bottlenecks in this centralized regime. Recursive estimators, such as PAGE, require periodic global full-gradient refreshes over all $nm$ samples, which are computationally expensive. Conversely, single-loop methods, such as SILVER, avoid such refreshes but require an impractical $\mathcal{O}(nm)$ memory footprint to store a control variate for every sample. In this paper, we propose SILAGE, a variance-reduced algorithm that addresses this trade-off. By actively exploiting the double-sum structure, SILAGE eliminates periodic global full-gradient refreshes over all $nm$ components (evaluating at most one local group gradient per iteration) while requiring only $\mathcal{O}(n)$ memory. Furthermore, we provide a tight convergence analysis that avoids pessimistic worst-case Lipschitz constants. Instead, SILAGE's complexity natively adapts to the underlying data geometry via nested functional similarities: across-group ($δ_1$) and within-group ($δ_2$) heterogeneity. Our results improve existing state-of-the-art bounds in several practically relevant regimes.

翻译：大规模数据集上的经验风险最小化自然呈现出嵌套双重有限和结构，其中$N=nm$个样本按逻辑或物理方式划分为$n$个大小为$m$的区块（例如在数据池化、外核学习或分层抽样场景中）。尽管方差缩减方法在非凸目标上实现了最优预言机复杂度，但在这种集中式场景中存在严重的扩展瓶颈。递归估计器（如PAGE）需要周期性对所有$nm$个样本进行全局全梯度刷新，计算开销高昂。相反，单循环方法（如SILVER）虽然避免了此类刷新，但需为每个样本存储控制变量，导致不可行的$\mathcal{O}(nm)$内存占用。本文提出SILAGE算法，通过主动利用双重和结构解决了这一权衡问题：该算法仅需$\mathcal{O}(n)$内存，且无需对所有$nm$个分量执行周期性全局全梯度刷新（每次迭代最多评估一个局部组梯度）。此外，我们给出了避免悲观最坏情况Lipschitz常数的紧致收敛性分析。SILAGE的复杂度通过嵌套函数相似性（组间异质性$\delta_1$与组内异质性$\delta_2$）自适应于底层数据几何特征。我们的结果在多个实际相关场景下改进了现有最优界。