Adapting pretrained language models to novel domains, such as clinical applications, traditionally involves retraining their entire set of parameters. However, this approach is increasingly proven to be impractical owing to the substantial computational requirements associated with training such large language models. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) techniques offer a viable solution by selectively fine-tuning a small subset of additional parameters, significantly reducing the computational requirements for domain adaptation. In this study, we propose Clinical LLaMA-LoRA, a PEFT adapter layer built upon the open-sourced LLaMA model. Clinical LLaMA-LoRA is trained using clinical notes obtained from the MIMIC-IV database, thereby creating a specialised adapter designed for the clinical domain. Additionally, we propose a two-step PEFT framework which fuses Clinical LLaMA-LoRA with Downstream LLaMA-LoRA, another PEFT adapter specialised for downstream tasks. We evaluate this framework on multiple clinical outcome prediction datasets, comparing it to clinically trained language models. Our proposed framework achieves a state-of-the-art AUROC score averaged across all clinical downstream tasks. We observe substantial improvements of 6-9% AUROC score in the large-scale multilabel classification tasks, such as diagnoses and procedures classification.

翻译：将预训练语言模型适配至临床应用等新领域时，传统方法需要重新训练其全部参数。然而，由于训练此类大型语言模型需要大量计算资源，这种方法已被证明越来越不切实际。为解决该问题，参数高效微调技术通过选择性微调少量新增参数，显著降低领域适配的计算需求，提供了一种可行的解决方案。本研究提出Clinical LLaMA-LoRA——一种基于开源LLaMA模型构建的参数高效微调适配器层。该适配器采用MIMIC-IV数据库中的临床记录进行训练，从而创建专用于临床领域的适配器。此外，我们提出双阶段参数高效微调框架，将Clinical LLaMA-LoRA与专注于下游任务的另一参数高效微调适配器Downstream LLaMA-LoRA进行融合。我们基于多个临床结局预测数据集评估该框架，并与经过临床训练的语言模型进行对比。所提框架在全部临床下游任务中取得了平均AUROC评分的最优结果。在大规模多标签分类任务（如诊断与手术分类）中，我们观察到AUROC评分显著提升6%-9%。