Large Language Models have become the core architecture upon which most modern natural language processing (NLP) systems build. These models can consistently deliver impressive accuracy and robustness across tasks and domains, but their high computational overhead can make inference difficult and expensive. To make using these models less costly, recent work has explored leveraging structured and unstructured pruning, quantization, and distillation to improve inference speed and decrease size. This paper studies how models pruned using Gradual Unstructured Magnitude Pruning can transfer between domains and tasks. Our experimentation shows that models that are pruned during pretraining using general domain masked language models can transfer to novel domains and tasks without extensive hyperparameter exploration or specialized approaches. We demonstrate that our general sparse model Sparse*BERT can become SparseBioBERT simply by pretraining the compressed architecture on unstructured biomedical text. Moreover, we show that SparseBioBERT can match the quality of BioBERT with only 10\% of the parameters.

翻译：大型语言模型已成为大多数现代自然语言处理系统的核心架构。这些模型能够在不同任务与领域间持续展现卓越的准确性与鲁棒性，但高昂的计算开销使得推理过程既困难又昂贵。为降低使用这些模型的成本，近期研究探索了利用结构化与非结构化剪枝、量化及蒸馏技术来提升推理速度并减小模型体积。本文研究采用渐进式非结构化幅度剪枝的模型如何在领域与任务间实现迁移。实验表明：在预训练阶段基于通用领域掩码语言模型进行剪枝的模型，无需大量超参数探索或专门方法即可迁移至新领域与新任务。我们证明，通用稀疏模型Sparse*BERT仅需对压缩架构进行非结构化生物医学文本预训练，即可转化为SparseBioBERT。更关键的是，SparseBioBERT仅用BioBERT 10%的参数即可达到与之相当的质量水平。