Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.

翻译：预先培训的语言模型(PLM)是大多数自然语言处理(NLP)任务的实际范例,也有益于生物医学领域:信息学、医学和计算机科学(CS)社区的研究人员提出各种经过生物医学数据集培训的PLM模型,例如生物医学文本、电子健康记录、蛋白质和DNA序列,用于生物医学任务;然而,生物医学PLM的跨纪律特征阻碍了生物医学模型在社区之间的传播;一些现有的工程相互分离,没有进行全面的比较和讨论;它期望进行一项调查,不仅系统地审查生物医学PLM及其应用的最新进展,而且将术语和基准标准化;在本文件中,我们总结生物医学领域经过培训的语言模型的最新进展及其在生物医学下游任务中的应用。我们特别讨论了生物医学生物医学模型的动机和分类,对生物医学下游任务的应用问题进行了详尽的讨论。最后,我们举例说明了各种限制和未来趋势,我们希望这些限制和趋势能够为研究界今后的研究提供启发。