Model compression has become the de-facto approach for optimizing the efficiency of vision models. Recently, the focus of most compression efforts has shifted to post-training scenarios due to the very high cost of large-scale pretraining. This has created the need to build compressible models from scratch, which can effectively be compressed after training. In this work, we present a sharpness-minimizing network transformation (SNT) method applied during pretraining that can create models with desirable compressibility and generalizability features. We compare our approach to a well-known sharpness-minimizing optimizer to validate its efficacy in creating a flat loss landscape. To the best of our knowledge, SNT is the first pretraining method that uses an architectural transformation to generate compression-friendly networks. We find that SNT generalizes across different compression tasks and network backbones, delivering consistent improvements over the ADAM baseline with up to 2% accuracy improvement on weight pruning and 5.4% accuracy improvement on quantization. Code to reproduce our results will be made publicly available.

翻译：模型压缩已成为优化视觉模型效率的事实标准。由于大规模预训练成本极高，近期多数压缩工作的重心已转向训练后场景。这催生了从零构建可压缩模型的需求，这类模型在训练后能被有效压缩。本文提出一种在预训练阶段应用的锐度最小化网络变换（SNT）方法，可生成兼具优良可压缩性与泛化性的模型。我们将该方法与著名的锐度最小化优化器进行对比，验证其在创建平坦损失景观方面的有效性。据我们所知，SNT是首个利用架构变换生成压缩友好型网络的预训练方法。我们发现SNT在不同压缩任务与网络骨干上具有泛化能力，相较于ADAM基线可实现持续提升：权重重剪任务准确率提升最高达2%，量化任务准确率提升最高达5.4%。用于复现结果的代码将公开发布。