The field of transfer learning is undergoing a significant shift with the introduction of large pretrained models which have demonstrated strong adaptability to a variety of downstream tasks. However, the high computational and memory requirements to finetune or use these models can be a hindrance to their widespread use. In this study, we present a solution to this issue by proposing a simple yet effective way to trade computational efficiency for asymptotic performance which we define as the performance a learning algorithm achieves as compute tends to infinity. Specifically, we argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance. We evaluate our method on the Nevis'22 continual learning benchmark that offers a diverse set of transfer scenarios. Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.

翻译：迁移学习领域正经历着重大变革，随着大规模预训练模型的引入，这些模型展现出对多种下游任务的强大适应性。然而，微调或使用这些模型所需的高计算和内存成本可能阻碍其广泛应用。在本研究中，我们提出了一种简单而有效的解决方案，通过权衡计算效率与渐近性能（即学习算法在计算资源趋于无穷时达到的性能）来应对这一问题。具体而言，我们论证了对预训练模型进行零样本结构化剪枝，能够在性能损失极小的情况下提升计算效率。我们在Nevis'22持续学习基准（该基准提供了多样化的迁移场景）上评估了我们的方法。结果表明，对预训练模型的卷积滤波器进行剪枝，可在低计算资源条件下带来超过20%的性能提升。