Recent works have explored the use of weight sparsity to improve the training efficiency (test accuracy w.r.t training FLOPs) of deep neural networks (DNNs). These works aim to reduce training FLOPs but training with sparse weights often leads to accuracy loss or requires longer train schedules, making the resulting training efficiency less clear. In contrast, we focus on using sparsity to increase accuracy while using the same FLOPS as the dense model and show training efficiency gains through higher accuracy. In this work, we introduce SIFT, a family of Sparse Iso-FLOP Transformations which are used as drop-in replacements for dense layers to improve their representational capacity and FLOP efficiency. Each transformation is parameterized by a single parameter (sparsity level) and provides a larger search space to find optimal sparse masks. Without changing any training hyperparameters, replacing dense layers with SIFT leads to significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL), both matching larger dense model variants with 2x or more FLOPs. To the best of our knowledge, this is the first work to demonstrate the use of sparsity for improving accuracy of dense models via a simple-to-use set of sparse transformations. Code is available at: https://github.com/CerebrasResearch/SIFT.

翻译：近期研究探索了利用权重稀疏性提升深度神经网络（DNN）的训练效率（即测试精度相对于训练FLOPs的指标）。这些工作旨在减少训练FLOPs，但稀疏权重训练常导致精度损失或需要更长的训练周期，使最终训练效率难以明确评估。与之相反，我们聚焦于利用稀疏性在保持与稠密模型相同FLOPs的前提下提升精度，并通过更高精度展现训练效率增益。本文提出SIFT（Sparse Iso-FLOP Transformations）系列方法，作为稠密层的即插即用替代方案，以提升其表征能力与FLOP效率。每种变换仅由单一参数（稀疏度）控制，为寻找最优稀疏掩码提供了更大搜索空间。在不改变任何训练超参数的情况下，将稠密层替换为SIFT在计算机视觉（CV）与自然语言处理（NLP）任务中均取得了显著提升，包括ImageNet上的ResNet-18（+3.5%）与WikiText-103上的GPT-3 Small（-0.4 PPL），其性能可媲美FLOPs翻倍甚至更高的更大规模稠密模型变体。据我们所知，这是首个通过简单易用的稀疏变换集证明稀疏性可提升稠密模型精度的研究工作。代码开源地址：https://github.com/CerebrasResearch/SIFT。