With the tremendous success of large transformer models in natural language understanding, down-sizing them for cost-effective deployments has become critical. Recent studies have explored the low-rank weight factorization techniques which are efficient to train, and apply out-of-the-box to any transformer architecture. Unfortunately, the low-rank assumption tends to be over-restrictive and hinders the expressiveness of the compressed model. This paper proposes, DSFormer, a simple alternative factorization scheme which expresses a target weight matrix as the product of a small dense and a semi-structured sparse matrix. The resulting approximation is more faithful to the weight distribution in transformers and therefore achieves a stronger efficiency-accuracy trade-off. Another concern with existing factorizers is their dependence on a task-unaware initialization step which degrades the accuracy of the resulting model. DSFormer addresses this issue through a novel Straight-Through Factorizer (STF) algorithm that jointly learns all the weight factorizations to directly maximize the final task accuracy. Extensive experiments on multiple natural language understanding benchmarks demonstrate that DSFormer obtains up to 40% better compression than the state-of-the-art low-rank factorizers, leading semi-structured sparsity baselines and popular knowledge distillation approaches. Our approach is also orthogonal to mainstream compressors and offers up to 50% additional compression when added to popular distilled, layer-shared and quantized transformers. We empirically evaluate the benefits of STF over conventional optimization practices.

翻译：随着大型Transformer模型在自然语言理解领域的巨大成功，如何缩减其规模以实现经济高效的部署变得至关重要。近期研究探索了低秩权重分解技术，该技术训练高效且可直接应用于任意Transformer架构。然而，低秩假设往往过于严格，限制了压缩模型的表达能力。本文提出DSFormer，一种简单的替代分解方案，将目标权重矩阵表示为小型密集矩阵与半结构化稀疏矩阵的乘积。所得近似更忠实于Transformer中的权重分布，因此实现了更强的效率-精度平衡。现有分解器的另一个缺陷是其依赖于与任务无关的初始化步骤，这会降低最终模型的精度。DSFormer通过新颖的直通式因子化（STF）算法解决了这一问题，该算法联合学习所有权重分解，以直接最大化最终任务精度。在多个自然语言理解基准上的大量实验表明，DSFormer相比最先进的低秩分解器实现了高达40%的更好压缩效果，显著优于半结构化稀疏基线方法和流行的知识蒸馏方法。我们的方法与主流压缩器正交，当将其应用于流行的蒸馏、层共享和量化Transformer时，可额外提供高达50%的压缩率。我们通过实验评估了STF相较于传统优化实践的优势。