We characterize the squared prediction risk of ensemble estimators obtained through subagging (subsample bootstrap aggregating) regularized M-estimators and construct a consistent estimator for the risk. Specifically, we consider a heterogeneous collection of $M \ge 1$ regularized M-estimators, each trained with (possibly different) subsample sizes, convex differentiable losses, and convex regularizers. We operate under the proportional asymptotics regime, where the sample size $n$, feature size $p$, and subsample sizes $k_m$ for $m \in [M]$ all diverge with fixed limiting ratios $n/p$ and $k_m/n$. Key to our analysis is a new result on the joint asymptotic behavior of correlations between the estimator and residual errors on overlapping subsamples, governed through a (provably) contractive nonlinear system of equations. Of independent interest, we also establish convergence of trace functionals related to degrees of freedom in the non-ensemble setting (with $M = 1$) along the way, extending previously known cases for squared loss with ridge and lasso regularizers. When specialized to homogeneous ensembles trained with a common loss, regularizer, and subsample size, the risk characterization sheds some light on the implicit regularization effect due to the ensemble and subsample sizes $(M,k)$. For any ensemble size $M$, optimally tuning subsample size yields sample-wise monotonic risk. For the full-ensemble estimator (when $M \to \infty$), the optimal subsample size $k^\star$ tends to be in the overparameterized regime $(k^\star \le \min\{n,p\})$, when explicit regularization is vanishing. Finally, joint optimization of subsample size, ensemble size, and regularization can significantly outperform regularizer optimization alone on the full data (without any subagging).

翻译：我们刻画了通过子袋法（子样本自助聚合）正则化M估计量得到的集成估计量的平方预测风险，并构建了风险的一致估计量。具体而言，我们考虑一个包含$M \ge 1$个正则化M估计量的异质集合，每个估计量使用（可能不同的）子样本大小、凸可微损失函数和凸正则化器进行训练。我们在比例渐近体系下进行分析，其中样本量$n$、特征量$p$以及子样本量$k_m$（$m \in [M]$）均以固定的极限比率$n/p$和$k_m/n$发散。我们分析的关键是关于重叠子样本上估计量与残差误差之间相关性的联合渐近行为的新结果，该结果由一个（可证明的）压缩非线性方程组控制。作为独立贡献，我们还在此过程中建立了非集成设置（$M = 1$）中与自由度相关的迹泛函的收敛性，扩展了先前已知的具有岭和lasso正则化器的平方损失情况。当专门针对使用共同损失函数、正则化器和子样本大小训练的同质集成时，风险刻画揭示了由于集成和子样本大小$(M,k)$所产生的隐式正则化效应。对于任何集成大小$M$，最优调整子样本大小会产生样本单调风险。对于全集成估计量（当$M \to \infty$时），当显式正则化趋于消失时，最优子样本大小$k^\star$倾向于处于过参数化区域（$k^\star \le \min\{n,p\}$）。最后，联合优化子样本大小、集成大小和正则化可以显著优于仅在全数据上优化正则化器（不进行任何子袋法）的性能。