Deep neural network ensembles combine the wisdom of multiple deep neural networks to improve the generalizability and robustness over individual networks. It has gained increasing popularity to study deep ensemble techniques in the deep learning community. Some mission-critical applications utilize a large number of deep neural networks to form deep ensembles to achieve desired accuracy and resilience, which introduces high time and space costs for ensemble execution. However, it still remains a critical challenge whether a small subset of the entire deep ensemble can achieve the same or better generalizability and how to effectively identify these small deep ensembles for improving the space and time efficiency of ensemble execution. This paper presents a novel deep ensemble pruning approach, which can efficiently identify smaller deep ensembles and provide higher ensemble accuracy than the entire deep ensemble of a large number of member networks. Our hierarchical ensemble pruning approach (HQ) leverages three novel ensemble pruning techniques. First, we show that the focal diversity metrics can accurately capture the complementary capacity of the member networks of an ensemble, which can guide ensemble pruning. Second, we design a focal diversity based hierarchical pruning approach, which will iteratively find high quality deep ensembles with low cost and high accuracy. Third, we develop a focal diversity consensus method to integrate multiple focal diversity metrics to refine ensemble pruning results, where smaller deep ensembles can be effectively identified to offer high accuracy, high robustness and high efficiency. Evaluated using popular benchmark datasets, we demonstrate that the proposed hierarchical ensemble pruning approach can effectively identify high quality deep ensembles with better generalizability while being more time and space efficient in ensemble decision making.

翻译：深度神经网络集成通过融合多个深度神经网络的智慧，相比单个网络能显著提升泛化能力和鲁棒性。近年来，深度学习领域对集成技术的研究日益深入。部分关键任务应用采用大量深度神经网络构成深度集成，以实现所需的准确性与弹性，但这带来了高昂的时间与空间开销。然而，能否从整个深度集成中选取一个小子集实现相同或更优的泛化能力，以及如何有效识别这些小型深度集成以提升空间与时间效率，仍是亟待解决的关键难题。本文提出一种新型深度集成剪枝方法，可高效识别更精简的深度集成，且其集成精度超越包含大量成员网络的完整深度集成。我们的层次化集成剪枝方法（HQ）融合了三种创新技术：首先，证明焦点多样性度量能精确捕捉集成中成员网络的互补能力，从而指导剪枝过程；其次，设计基于焦点多样性的层次化剪枝策略，以低成本迭代获取高质量、高精度的深度集成；最后，开发焦点多样性共识方法，整合多种焦点多样性度量以优化剪枝结果，有效识别兼具高精度、强鲁棒性与高效率的小型深度集成。通过主流基准数据集的评估，本方法不仅能有效识别高质量深度集成，实现更优泛化能力，同时显著提升集成决策的时间与空间效率。