Interpolators are unstable. For example, the mininum $\ell_2$ norm least square interpolator exhibits unbounded test errors when dealing with noisy data. In this paper, we study how ensemble stabilizes and thus improves the generalization performance, measured by the out-of-sample prediction risk, of an individual interpolator. We focus on bagged linear interpolators, as bagging is a popular randomization-based ensemble method that can be implemented in parallel. We introduce the multiplier-bootstrap-based bagged least square estimator, which can then be formulated as an average of the sketched least square estimators. The proposed multiplier bootstrap encompasses the classical bootstrap with replacement as a special case, along with a more intriguing variant which we call the Bernoulli bootstrap. Focusing on the proportional regime where the sample size scales proportionally with the feature dimensionality, we investigate the out-of-sample prediction risks of the sketched and bagged least square estimators in both underparametrized and overparameterized regimes. Our results reveal the statistical roles of sketching and bagging. In particular, sketching modifies the aspect ratio and shifts the interpolation threshold of the minimum $\ell_2$ norm estimator. However, the risk of the sketched estimator continues to be unbounded around the interpolation threshold due to excessive variance. In stark contrast, bagging effectively mitigates this variance, leading to a bounded limiting out-of-sample prediction risk. To further understand this stability improvement property, we establish that bagging acts as a form of implicit regularization, substantiated by the equivalence of the bagged estimator with its explicitly regularized counterpart. We also discuss several extensions.

翻译：插值器是不稳定的。例如，在处理含噪声数据时，最小 $\ell_2$ 范数最小二乘插值器会表现出无界的测试误差。本文研究集成如何稳定并从而提升单个插值器的泛化性能，我们通过样本外预测风险来衡量。我们聚焦于装袋线性插值器，因为装袋是一种流行的基于随机化的集成方法，可并行实现。我们引入了基于乘数自助法的装袋最小二乘估计器，该估计器可表述为草图最小二乘估计器的平均值。所提出的乘数自助法包含经典的有放回自助法作为特例，以及一个更有趣的变体，我们称之为伯努利自助法。聚焦于样本规模与特征维度成比例增长的机制，我们研究了欠参数化和过参数化两种情况下草图最小二乘估计器和装袋最小二乘估计器的样本外预测风险。我们的结果揭示了草图和装袋的统计作用。特别地，草图改变了最小 $\ell_2$ 范数估计器的纵横比并移动了插值阈值。然而，由于过大的方差，草图估计器的风险在插值阈值附近仍然是无界的。与此形成鲜明对比的是，装袋有效缓解了这一方差，导致样本外预测风险有界。为进一步理解这种稳定性提升的性质，我们证明了装袋作为一种隐式正则化形式，这一结论由装袋估计器与其显式正则化对应物之间的等价性所支持。我们还讨论了几种扩展。