We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.

翻译：我们正式研究了深度学习模型的集成如何提高测试准确率，以及如何通过知识蒸馏将集成的优越性能萃取到单个模型中。我们考虑具有挑战性的场景：集成仅由几个独立训练的、使用相同架构的神经网络输出的平均值构成，这些网络使用相同算法和相同数据集训练，仅因初始化所用的随机种子不同而有所差异。我们证明，深度学习中的集成/知识蒸馏与传统学习理论（如Boosting或神经正切核NTKs）运作方式截然不同。为正确理解这些机制，我们提出了一项理论：当数据具有我们称为“多视角”的结构时，独立训练的神经网络集成能显著提升测试准确率，并且这种优越的测试准确率可通过训练单个模型匹配集成输出（而非真实标签）而被可靠地蒸馏到单一模型中。我们的研究揭示了深度学习集成以完全不同于传统定理的方式运作的机理，并阐明了“暗知识”如何隐藏于集成输出中并用于蒸馏。最后，我们证明，自蒸馏可被视为隐式结合集成与知识蒸馏来提升测试准确率的方法。