Adapting pretrained language models to novel domains, such as clinical applications, traditionally involves retraining their entire set of parameters. However, this approach is increasingly proven to be impractical owing to the substantial computational requirements associated with training such large language models. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) techniques offer a viable solution by selectively fine-tuning a small subset of additional parameters, significantly reducing the computational requirements for domain adaptation. In this study, we propose Clinical LLaMA-LoRA, a PEFT adapter layer built upon the open-sourced LLaMA model. Clinical LLaMA-LoRA is trained using clinical notes obtained from the MIMIC-IV database, thereby creating a specialised adapter designed for the clinical domain. Additionally, we propose a two-step PEFT framework which fuses Clinical LLaMA-LoRA with Downstream LLaMA-LoRA, another PEFT adapter specialised for downstream tasks. We evaluate this framework on multiple clinical outcome prediction datasets, comparing it to clinically trained language models. Our proposed framework achieves a state-of-the-art AUROC score averaged across all clinical downstream tasks. We observe substantial improvements of 6-9% AUROC score in the large-scale multilabel classification tasks, such as diagnoses and procedures classification.

翻译：将预训练语言模型适配到临床应用等新领域时，传统方法需重新训练全部参数。然而，由于训练此类大型语言模型需要巨大的计算资源，这种方法已被证实越来越不切实际。为解决该问题，参数高效微调技术通过选择性地微调少量额外参数提供了一种可行方案，显著降低了领域适配的计算需求。本研究提出了Clinical LLaMA-LoRA，一种基于开源LLaMA模型构建的参数高效微调适配器层。Clinical LLaMA-LoRA使用MIMIC-IV数据库中的临床笔记进行训练，从而创建专用于临床领域的适配器。此外，我们提出了一种两阶段参数高效微调框架，将Clinical LLaMA-LoRA与另一个专用于下游任务的参数高效微调适配器Downstream LLaMA-LoRA进行融合。我们在多个临床结局预测数据集上评估该框架，并与临床训练语言模型进行比较。所提框架在所有临床下游任务的平均AUROC评分上达到了最优水平。在大规模多标签分类任务（如诊断和手术分类）中，我们观察到AUROC评分有6-9%的显著提升。