The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural Language Processing (NLP). Instead of directly training on a downstream task, language models are first pre-trained on large datasets with cross-domain knowledge (e.g., Pile, MassiveText, etc.) and then fine-tuned on task-specific data (e.g., natural language generation, text summarization, etc.). Scaling the model and dataset size has helped improve the performance of LLMs, but unfortunately, this also leads to highly prohibitive computational costs. Pre-training LLMs often require orders of magnitude more FLOPs than fine-tuning and the model capacity often remains the same between the two phases. To achieve training efficiency w.r.t training FLOPs, we propose to decouple the model capacity between the two phases and introduce Sparse Pre-training and Dense Fine-tuning (SPDF). In this work, we show the benefits of using unstructured weight sparsity to train only a subset of weights during pre-training (Sparse Pre-training) and then recover the representational capacity by allowing the zeroed weights to learn (Dense Fine-tuning). We demonstrate that we can induce up to 75% sparsity into a 1.3B parameter GPT-3 XL model resulting in a 2.5x reduction in pre-training FLOPs, without a significant loss in accuracy on the downstream tasks relative to the dense baseline. By rigorously evaluating multiple downstream tasks, we also establish a relationship between sparsity, task complexity, and dataset size. Our work presents a promising direction to train large GPT models at a fraction of the training FLOPs using weight sparsity while retaining the benefits of pre-trained textual representations for downstream tasks.

翻译：预训练-微调范式为自然语言处理领域带来了诸多突破性进展。语言模型并非直接在下游任务上训练，而是首先在包含跨领域知识的大规模数据集（如Pile、MassiveText等）上进行预训练，随后在特定任务数据（如自然语言生成、文本摘要等）上进行微调。扩大模型与数据集规模虽能提升大语言模型性能，但同时也带来了高昂的计算成本。相较于微调阶段，预训练大语言模型往往需要高出数个数量级的FLOPs，且两个阶段的模型容量通常保持不变。为提升训练FLOPs效率，我们提出解耦两个阶段的模型容量，并引入稀疏预训练与稠密微调（SPDF）方法。本研究展示了非结构化权重稀疏性的优势：在预训练阶段仅对部分权重子集进行训练（稀疏预训练），随后通过允许零权重学习来恢复表征能力（稠密微调）。实验证明，我们可在含13亿参数的GPT-3 XL模型中引入高达75%的稀疏性，使预训练FLOPs降低2.5倍，同时在下游任务精度上与稠密基线模型相比无显著损失。通过严格评估多个下游任务，我们还建立了稀疏性、任务复杂度与数据集规模之间的关联。本研究提供了一种具有前景的技术路径——在保留预训练文本表征对下游任务有益性的前提下，借助权重稀疏性以极低的训练FLOPs训练大型GPT模型。