Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.

翻译：预训练语言模型（PLMs）已成为大多数自然语言处理（NLP）任务的事实标准范式。这一趋势同样惠及生物医学领域：来自信息学、医学和计算机科学（CS）领域的研究人员提出了多种基于生物医学数据集（如生物医学文本、电子健康病历、蛋白质和DNA序列）训练的PLMs，用于各类生物医学任务。然而，生物医学PLMs的跨学科特性阻碍了其在各领域间的传播；现有部分研究工作彼此孤立，缺乏全面的比较与讨论。因此，亟需一篇综述，不仅系统梳理生物医学PLMs及其应用的最新进展，还需规范术语和基准。本文总结了生物医学领域预训练语言模型及其在生物医学下游任务中的最新研究进展。具体而言，我们探讨了现有生物医学PLMs的构建动机，并提出了分类体系；详尽讨论了其在生物医学下游任务中的应用。最后，我们阐述了当前存在的各种局限性与未来趋势，期望能为研究社区的未来探索提供启示。