Pruning is a widely used technique for reducing the size of deep neural networks while maintaining their performance. However, such a technique, despite being able to massively compress deep models, is hardly able to remove entire layers from a model (even when structured): is this an addressable task? In this study, we introduce EGP, an innovative Entropy Guided Pruning algorithm aimed at reducing the size of deep neural networks while preserving their performance. The key focus of EGP is to prioritize pruning connections in layers with low entropy, ultimately leading to their complete removal. Through extensive experiments conducted on popular models like ResNet-18 and Swin-T, our findings demonstrate that EGP effectively compresses deep neural networks while maintaining competitive performance levels. Our results not only shed light on the underlying mechanism behind the advantages of unstructured pruning, but also pave the way for further investigations into the intricate relationship between entropy, pruning techniques, and deep learning performance. The EGP algorithm and its insights hold great promise for advancing the field of network compression and optimization. The source code for EGP is released open-source.

翻译：剪枝是一种广泛用于减小深度神经网络规模同时保持其性能的技术。然而，尽管该技术能够大幅压缩深度模型，却几乎无法从模型中移除整个层（即使是结构化剪枝）：这一任务是否可以实现？在本研究中，我们提出了EGP，一种创新的基于熵引导的剪枝算法，旨在减小深度神经网络的规模同时保持其性能。EGP的核心重点在于优先剪枝低熵层中的连接，最终实现这些层的完全移除。通过在ResNet-18和Swin-T等流行模型上进行的大量实验，我们的结果表明，EGP能够有效压缩深度神经网络，同时保持具有竞争力的性能水平。我们的结果不仅揭示了非结构化剪枝优势背后的内在机制，还为深入探究熵、剪枝技术与深度学习性能之间的复杂关系铺平了道路。EGP算法及其见解对推动网络压缩与优化领域的发展具有重要前景。EGP的源代码已开源发布。