Neural networks can be significantly compressed by pruning, leading to sparse models requiring considerably less storage and floating-point operations while maintaining predictive performance. Model soups (Wortsman et al., 2022) improve generalization and out-of-distribution performance by averaging the parameters of multiple models into a single one without increased inference time. However, identifying models in the same loss basin to leverage both sparsity and parameter averaging is challenging, as averaging arbitrary sparse models reduces the overall sparsity due to differing sparse connectivities. In this work, we address these challenges by demonstrating that exploring a single retraining phase of Iterative Magnitude Pruning (IMP) with varying hyperparameter configurations, such as batch ordering or weight decay, produces models that are suitable for averaging and share the same sparse connectivity by design. Averaging these models significantly enhances generalization performance compared to their individual components. Building on this idea, we introduce Sparse Model Soups (SMS), a novel method for merging sparse models by initiating each prune-retrain cycle with the averaged model of the previous phase. SMS maintains sparsity, exploits sparse network benefits being modular and fully parallelizable, and substantially improves IMP's performance. Additionally, we demonstrate that SMS can be adapted to enhance the performance of state-of-the-art pruning during training approaches.

翻译：神经网络可通过剪枝实现显著压缩，生成稀疏模型，在保持预测性能的同时大幅减少存储需求与浮点运算量。模型汤（Wortsman等人，2022）通过将多个模型的参数平均化为单一模型来提升泛化能力与分布外性能，且不增加推理时间。然而，由于不同稀疏连接性会导致平均任意稀疏模型时降低整体稀疏度，如何识别位于相同损失盆地中的模型以兼顾稀疏性与参数平均颇具挑战。本研究通过论证以下发现解决了上述难题：在迭代幅度剪枝（IMP）的单次重训练阶段中，采用不同超参数配置（如批次顺序或权重衰减）探索生成的模型天然具备可平均性且共享相同的稀疏连接结构。平均这些模型相较于其个体组件能显著提升泛化性能。基于这一思想，我们提出稀疏模型汤（SMS），一种通过用上一阶段的平均模型初始化每次剪枝重训练周期来合并稀疏模型的新方法。SMS在保持稀疏性的同时，充分利用稀疏网络模块化且完全可并行的优势，显著提升了IMP的性能。此外，我们证明SMS可被适配用于增强训练过程中最先进剪枝方法的性能。