As opposed to general English, many concepts in biomedical terminology have been designed in recent history by biomedical professionals with the goal of being precise and concise. This is often achieved by concatenating meaningful biomedical morphemes to create new semantic units. Nevertheless, most modern biomedical language models (LMs) are pre-trained using standard domain-specific tokenizers derived from large scale biomedical corpus statistics without explicitly leveraging the agglutinating nature of biomedical language. In this work, we first find that standard open-domain and biomedical tokenizers are largely unable to segment biomedical terms into meaningful components. Therefore, we hypothesize that using a tokenizer which segments biomedical terminology more accurately would enable biomedical LMs to improve their performance on downstream biomedical NLP tasks, especially ones which involve biomedical terms directly such as named entity recognition (NER) and entity linking. Surprisingly, we find that pre-training a biomedical LM using a more accurate biomedical tokenizer does not improve the entity representation quality of a language model as measured by several intrinsic and extrinsic measures such as masked language modeling prediction (MLM) accuracy as well as NER and entity linking performance. These quantitative findings, along with a case study which explores entity representation quality more directly, suggest that the biomedical pre-training process is quite robust to instances of sub-optimal tokenization.

翻译：与通用英语不同，生物医学术语中的许多概念是由生物医学专业人士在近代史中以精确简洁为目标设计的。这通常通过将有意义的生物医学语素进行联结来创造新的语义单元实现。然而，大多数现代生物医学语言模型仍采用基于大规模生物医学语料库统计衍生出的标准领域专用分词器进行预训练，并未显式利用生物医学语言的黏着性特征。本研究发现，标准开放域分词器与生物医学分词器在将生物医学术语切分为有意义的语义成分方面存在显著缺陷。因此我们假设：若采用更精准切分生物医学术语的分词器，将提升生物医学语言模型在下游生物医学自然语言处理任务（特别是直接涉及术语的任务如命名实体识别和实体链接）中的表现。令人意外的是，实验结果表明：使用更精确的生物医学分词器预训练语言模型，并未提升模型的实体表征质量——这通过掩码语言模型预测准确率、命名实体识别及实体链接等多种内在与外在评估指标得以验证。这些量化发现结合探索实体表征质量的案例研究表明，生物医学预训练过程对次优分词实例具有相当强的鲁棒性。