As language models increase in size by the day, methods for efficient inference are critical to leveraging their capabilities for various applications. Prior work has investigated techniques like model pruning, knowledge distillation, and data multiplexing to increase model throughput without sacrificing accuracy. In this paper, we combine two such methods -- structured pruning and data multiplexing -- to compound the speedup gains obtained by either method. Our approach, PruMUX, obtains up to 7.5-29.5X throughput improvement over BERT-base model with accuracy threshold from 80% to 74%. We further study various combinations of parameters (such as sparsity and multiplexing factor) in the two techniques to provide a comprehensive analysis of the tradeoff between accuracy and throughput in the resulting models. We then propose Auto-PruMUX, a meta-level model that can predict the high-performance parameters for pruning and multiplexing given a desired accuracy loss budget, providing a practical method to leverage the combination effectively.

翻译：随着语言模型规模日益增长，高效推理方法对于发挥其在各类应用中的能力至关重要。已有研究探讨了模型剪枝、知识蒸馏及数据复用等技术，以期在不牺牲准确率的前提下提升模型吞吐量。本文融合了结构化剪枝与数据复用两种方法，以复合增强单一方法带来的加速增益。我们提出的PruMUX方法在80%至74%准确率阈值下，相比BERT-base模型实现了7.5至29.5倍的吞吐量提升。进一步地，我们系统研究了两种技术中多种参数组合（如稀疏度与复用因子），以全面分析所得模型在准确率与吞吐量之间的权衡关系。最终，我们提出了Auto-PruMUX元模型，该模型能够根据给定的准确率损失预算，预测剪枝与复用的高性能参数，为有效结合这两种方法提供了实用方案。