Ensembling is among the most popular tools in machine learning (ML) due to its effectiveness in minimizing variance and thus improving generalization. Most ensembling methods for black-box base learners fall under the umbrella of "stacked generalization," namely training an ML algorithm that takes the inferences from the base learners as input. While stacking has been widely applied in practice, its theoretical properties are poorly understood. In this paper, we prove a novel result, showing that choosing the best stacked generalization from a (finite or finite-dimensional) family of stacked generalizations based on cross-validated performance does not perform "much worse" than the oracle best. Our result strengthens and significantly extends the results in Van der Laan et al. (2007). Inspired by the theoretical analysis, we further propose a particular family of stacked generalizations in the context of probabilistic forecasting, each one with a different sensitivity for how much the ensemble weights are allowed to vary across items, timestamps in the forecast horizon, and quantiles. Experimental results demonstrate the performance gain of the proposed method.

翻译：集成学习是机器学习（ML）中最受欢迎的工具之一，因其在减少方差从而提升泛化能力方面的有效性而备受青睐。大多数针对黑箱基学习器的集成方法都属于“堆叠泛化”的范畴，即训练一个以基学习器的推断结果为输入的机器学习算法。尽管堆叠法在实践中被广泛应用，但其理论性质尚不明确。本文证明了一个新颖的结果：基于交叉验证性能从（有限或有限维）堆叠泛化族中选择最优堆叠泛化，其表现不会比理论最优解“差很多”。我们的结果强化并显著扩展了Van der Laan等人（2007）的研究。受理论分析的启发，我们进一步在概率预测的背景下提出了一类特定的堆叠泛化族，其中每个成员对集成权重在项目、预测时间戳和分位数间的变化敏感度各不相同。实验结果表明了所提方法的性能提升。