Specialised pre-trained language models are becoming more frequent in NLP since they can potentially outperform models trained on generic texts. BioBERT and BioClinicalBERT are two examples of such models that have shown promise in medical NLP tasks. Many of these models are overparametrised and resource-intensive, but thanks to techniques like Knowledge Distillation (KD), it is possible to create smaller versions that perform almost as well as their larger counterparts. In this work, we specifically focus on development of compact language models for processing clinical texts (i.e. progress notes, discharge summaries etc). We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning, with the number of parameters ranging from 15 million to 65 million. These models performed comparably to larger models such as BioBERT and ClinicalBioBERT and significantly outperformed other compact models trained on general or biomedical data. Our extensive evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks, including Natural Language Inference, Relation Extraction, Named Entity Recognition, and Sequence Classification. To our knowledge, this is the first comprehensive study specifically focused on creating efficient and compact transformers for clinical NLP tasks. The models and code used in this study can be found on our Huggingface profile at https://huggingface.co/nlpie and Github page at https://github.com/nlpie-research/Lightweight-Clinical-Transformers, respectively, promoting reproducibility of our results.

翻译：专业化预训练语言模型在自然语言处理领域日益常见，因其在通用文本训练模型基础上可能展现出更优性能。BioBERT与BioClinicalBERT即为在医学NLP任务中展现潜力的典范。尽管此类模型常存在参数冗余与资源消耗大的问题，但借助知识蒸馏（KD）等技术，可构建性能接近其大型版本的轻量化模型。本研究聚焦于开发适用于临床文本（如病程记录、出院小结等）处理的紧凑型语言模型。通过知识蒸馏与持续学习策略，我们开发了多个高效的轻量化临床Transformer模型，参数量范围从1500万至6500万。这些模型在性能上与BioBERT、ClinicalBioBERT等大型模型相当，且显著优于基于通用或生物医学数据训练的其他紧凑型模型。我们在多个标准数据集上开展了全面评估，涵盖自然语言推理、关系抽取、命名实体识别及序列分类等临床文本挖掘任务。据我们所知，这是首项专门针对临床NLP任务构建高效紧凑型Transformer模型的系统性研究。本研究所用模型与代码分别发布在Huggingface平台（https://huggingface.co/nlpie）和Github仓库（https://github.com/nlpie-research/Lightweight-Clinical-Transformers），以促进结果的可复现性。