Pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT) have achieved state-of-the-art performance in natural language processing (NLP) tasks. Recently, BERT has been adapted to the biomedical domain. Despite the effectiveness, these models have hundreds of millions of parameters and are computationally expensive when applied to large-scale NLP applications. We hypothesized that the number of parameters of the original BERT can be dramatically reduced with minor impact on performance. In this study, we present Bioformer, a compact BERT model for biomedical text mining. We pretrained two Bioformer models (named Bioformer8L and Bioformer16L) which reduced the model size by 60% compared to BERTBase. Bioformer uses a biomedical vocabulary and was pre-trained from scratch on PubMed abstracts and PubMed Central full-text articles. We thoroughly evaluated the performance of Bioformer as well as existing biomedical BERT models including BioBERT and PubMedBERT on 15 benchmark datasets of four different biomedical NLP tasks: named entity recognition, relation extraction, question answering and document classification. The results show that with 60% fewer parameters, Bioformer16L is only 0.1% less accurate than PubMedBERT while Bioformer8L is 0.9% less accurate than PubMedBERT. Both Bioformer16L and Bioformer8L outperformed BioBERTBase-v1.1. In addition, Bioformer16L and Bioformer8L are 2-3 fold as fast as PubMedBERT/BioBERTBase-v1.1. Bioformer has been successfully deployed to PubTator Central providing gene annotations over 35 million PubMed abstracts and 5 million PubMed Central full-text articles. We make Bioformer publicly available via https://github.com/WGLab/bioformer, including pre-trained models, datasets, and instructions for downstream use.

翻译：诸如双向编码器表示（BERT）等预训练语言模型已在自然语言处理（NLP）任务中取得最先进性能。近年来，BERT被适配至生物医学领域。尽管这些模型表现出色，但其拥有数亿参数，在应用于大规模NLP任务时计算成本高昂。我们假设原始BERT的参数数量可在不影响性能的前提下大幅缩减。本研究提出了Bioformer——一种用于生物医学文本挖掘的紧凑型BERT模型。我们预训练了两个Bioformer模型（分别命名为Bioformer8L和Bioformer16L），其模型体积相较BERTBase缩减了60%。Bioformer采用生物医学词汇表，并在PubMed摘要与PubMed Central全文文献上从头进行预训练。我们针对四项不同生物医学NLP任务（命名实体识别、关系抽取、问答系统与文档分类）的15个基准数据集，全面评估了Bioformer及现有生物医学BERT模型（包括BioBERT和PubMedBERT）的性能。结果表明，在参数减少60%的情况下，Bioformer16L的准确率仅比PubMedBERT低0.1%，而Bioformer8L则低0.9%。两种Bioformer模型均优于BioBERTBase-v1.1。此外，Bioformer16L和Bioformer8L的处理速度比PubMedBERT/BioBERTBase-v1.1快2-3倍。Bioformer已成功部署至PubTator Central，为超过3500万篇PubMed摘要与500万篇PubMed Central全文文献提供基因注释。我们通过https://github.com/WGLab/bioformer公开提供Bioformer，包括预训练模型、数据集及下游使用说明。