Recent works have explored the use of weight sparsity to improve the training efficiency (test accuracy w.r.t training FLOPs) of deep neural networks (DNNs). These works aim to reduce training FLOPs but training with sparse weights often leads to accuracy loss or requires longer training schedules, making the resulting training efficiency less clear. In contrast, we focus on using sparsity to increase accuracy while using the same FLOPs as the dense model and show training efficiency gains through higher accuracy. In this work, we introduce Sparse-IFT, a family of Sparse Iso-FLOP Transformations which are used as drop-in replacements for dense layers to improve their representational capacity and FLOP efficiency. Each transformation is parameterized by a single hyperparameter (sparsity level) and provides a larger search space to find optimal sparse masks. Without changing any training hyperparameters, replacing dense layers with Sparse-IFT leads to significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL), both matching larger dense model variants that use 2x or more FLOPs. To our knowledge, this is the first work to demonstrate the use of sparsity for improving the accuracy of dense models via a simple-to-use set of sparse transformations. Code is available at: https://github.com/CerebrasResearch/Sparse-IFT.

翻译：近期工作探索了利用权重稀疏性来提升深度神经网络（DNN）的训练效率（即测试准确率相对于训练FLOPs的度量）。这些工作旨在降低训练FLOPs，但稀疏权重训练常导致准确率下降或需要更长的训练计划，使得最终训练效率不够明确。与此不同，我们聚焦于在保持与稠密模型相同FLOPs的前提下，通过稀疏性提升准确率，并以更高准确率彰显训练效率增益。本文提出Sparse-IFT系列稀疏等FLOP变换（Sparse Iso-FLOP Transformations），可作为稠密层的即插即用替代，增强其表示能力与FLOP效率。每种变换由单超参数（稀疏度）参数化，提供更大搜索空间以寻找最优稀疏掩码。在无需更改任何训练超参数的情况下，将稠密层替换为Sparse-IFT可显著提升计算机视觉（CV）与自然语言处理（NLP）任务性能：包括ImageNet上的ResNet-18（+3.5%）与WikiText-103上的GPT-3 Small（-0.4 PPL），两者均匹配使用2倍或更多FLOPs的更大稠密模型变体。据我们所知，这是首个通过简单易用的稀疏变换集来证明利用稀疏性提升稠密模型准确率的工作。代码开源地址：https://github.com/CerebrasResearch/Sparse-IFT。