Objective: Clinical knowledge enriched transformer models (e.g., ClinicalBERT) have state-of-the-art results on clinical NLP (natural language processing) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (e.g., Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long clinical texts. Materials and Methods: Inspired by the success of long sequence transformer models and the fact that clinical notes are mostly long, we introduce two domain enriched language models, Clinical-Longformer and Clinical-BigBird, which are pre-trained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks. Results: The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT and other short-sequence transformers in all 10 downstream tasks and achieve new state-of-the-art results. Discussion: Our pre-trained language models provide the bedrock for clinical NLP using long texts. We have made our source code available at https://github.com/luoyuanlab/Clinical-Longformer, and the pre-trained models available for public download at: https://huggingface.co/yikuan8/Clinical-Longformer. Conclusion: This study demonstrates that clinical knowledge enriched long-sequence transformers are able to learn long-term dependencies in long clinical text. Our methods can also inspire the development of other domain-enriched long-sequence transformers.

翻译：目的：临床知识增强的Transformer模型（如ClinicalBERT）在临床自然语言处理任务中取得了最先进的结果。然而，这类Transformer模型的核心限制之一在于其完整的自注意力机制导致大量内存消耗，从而在长临床文本中性能下降。为解决这一问题，我们提出利用长序列Transformer模型（如Longformer和BigBird），将最大输入序列长度从512扩展到4096，以增强长临床文本中长期依赖关系的建模能力。材料与方法：受长序列Transformer模型成功经验及临床笔记大多为长文本这一事实的启发，我们引入了两个领域增强的语言模型——Clinical-Longformer和Clinical-BigBird，并在大规模临床语料库上进行预训练。我们使用10项基线任务（包括命名实体识别、问答、自然语言推理和文档分类）对这两个语言模型进行评估。结果：结果表明，Clinical-Longformer和Clinical-BigBird在所有10项下游任务中均持续且显著优于ClinicalBERT及其他短序列Transformer，并取得了新的最先进结果。讨论：我们的预训练语言模型为基于长文本的临床自然语言处理奠定了基础。相关源代码已开源在https://github.com/luoyuanlab/Clinical-Longformer，预训练模型可通过https://huggingface.co/yikuan8/Clinical-Longformer公开下载。结论：本研究表明，临床知识增强的长序列Transformer能够学习长临床文本中的长期依赖关系，我们的方法也为其他领域增强的长序列Transformer开发提供了启发。