Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we conduct an extensive empirical analysis of 12 language models, ranging from 220M to 175B parameters, measuring their performance on 3 different clinical tasks that test their ability to parse and reason over electronic health records. As part of our experiments, we train T5-Base and T5-Large models from scratch on clinical notes from MIMIC III and IV to directly investigate the efficiency of clinical tokens. We show that relatively small specialized clinical models substantially outperform all in-context learning approaches, even when finetuned on limited annotated data. Further, we find that pretraining on clinical tokens allows for smaller, more parameter-efficient models that either match or outperform much larger language models trained on general text. We release the code and the models used under the PhysioNet Credentialed Health Data license and data use agreement.

翻译：尽管近期大规模语言模型（LLMs）的扩展进展显著提升了诸多自然语言处理任务的性能，但尚不明确这类主要基于通用网络文本训练的模型是否适用于高度专业化、安全敏感的领域（如临床文本）。最新研究表明，LLMs编码了惊人的医学知识量，这引发了关于小型领域专用语言模型实用价值的重要问题：在通用域LLMs取得成功的背景下，是否仍有必要构建专业化临床模型？为探究此问题，我们对12个参数规模从2.2亿到1750亿的语言模型开展了广泛实证分析，在3项测试电子健康记录解析与推理能力的临床任务上评估其表现。在实验过程中，我们基于MIMIC III和IV临床病历从零训练了T5-Base和T5-Large模型，以直接探究临床词元的效率优势。结果表明，相对小型的专用临床模型在全部上下文学习方法中表现显著优越，即使仅使用有限的标注数据进行微调。进一步发现，基于临床词元的预训练能够构建更小、参数效率更高的模型，其性能可与甚至超越基于通用文本训练的更大规模语言模型。我们依据PhysioNet认证健康数据许可与数据使用协议发布所用代码与模型。