Large language models (LLMs) are typically trained on general source data for various domains, but a recent surge in domain-specific LLMs has shown their potential to outperform general-purpose models in domain-specific tasks (e.g., biomedicine). Although domain-specific pre-training enhances efficiency and leads to smaller models, the computational costs of training these LLMs remain high, posing budgeting challenges. We introduce MediSwift, a suite of biomedical LMs that leverage sparse pre-training on domain-specific biomedical text data. By inducing up to 75% weight sparsity during the pre-training phase, MediSwift achieves a 2-2.5x reduction in training FLOPs. Notably, all sparse pre-training was performed on the Cerebras CS-2 system, which is specifically designed to realize the acceleration benefits from unstructured weight sparsity, thereby significantly enhancing the efficiency of the MediSwift models. Through subsequent dense fine-tuning and strategic soft prompting, MediSwift models outperform existing LLMs up to 7B parameters on biomedical tasks, setting new benchmarks w.r.t efficiency-accuracy on tasks such as PubMedQA. Our results show that sparse pre-training, along with dense fine-tuning and soft prompting, offers an effective method for creating high-performing, computationally efficient models in specialized domains.

翻译：大型语言模型通常基于通用源数据进行多领域训练，但近期领域专用模型的兴起表明，其在特定领域任务（如生物医学）中可能超越通用模型。尽管领域专用预训练能提升效率并缩小模型规模，但训练这类语言模型的计算成本依然高昂，给预算带来挑战。我们提出MediSwift——一套利用生物医学领域文本数据进行稀疏预训练的生物医学语言模型。通过在预训练阶段引入高达75%的权重稀疏性，MediSwift实现了训练浮点运算次数降低2至2.5倍。值得注意的是，所有稀疏预训练均在Cerebras CS-2系统上完成，该系统专为利用非结构化权重稀疏性加速设计，从而显著提升MediSwift模型的效率。通过后续的密集微调与策略性软提示，MediSwift模型在PubMedQA等生物医学任务上以高达70亿参数超越现有语言模型，在效率与准确度方面树立了新标杆。我们的结果表明，稀疏预训练结合密集微调与软提示，为在专业领域构建高性能、计算高效的模型提供了有效方法。