Stacking, a heuristic technique for training deep residual networks by progressively increasing the number of layers and initializing new layers by copying parameters from older layers, has proven quite successful in improving the efficiency of training deep neural networks. In this paper, we propose a theoretical explanation for the efficacy of stacking: viz., stacking implements a form of Nesterov's accelerated gradient descent. The theory also covers simpler models such as the additive ensembles constructed in boosting methods, and provides an explanation for a similar widely-used practical heuristic for initializing the new classifier in each round of boosting. We also prove that for certain deep linear residual networks, stacking does provide accelerated training, via a new potential function analysis of the Nesterov's accelerated gradient method which allows errors in updates. We conduct proof-of-concept experiments to validate our theory as well.

翻译：堆积是一种通过逐层增加深度残差网络层数、并复制旧层参数初始化新层的启发式训练技术，已被证明能有效提升深度神经网络训练效率。本文提出堆积有效性的一种理论解释：即堆积实现了涅斯捷罗夫加速梯度下降的一种形式。该理论同样适用于提升方法中构建的加性集成等更简单模型，并为提升算法每轮新分类器初始化中广泛使用的类似实用启发式方法提供了理论依据。我们还证明，针对特定深度线性残差网络，通过允许更新存在误差的涅斯捷罗夫加速梯度法新势函数分析，堆积确实能实现加速训练。最后通过概念验证实验对理论进行验证。