As opposed to general English, many concepts in biomedical terminology have been designed in recent history by biomedical professionals with the goal of being precise and concise. This is often achieved by concatenating meaningful biomedical morphemes to create new semantic units. Nevertheless, most modern biomedical language models (LMs) are pre-trained using standard domain-specific tokenizers derived from large scale biomedical corpus statistics without explicitly leveraging the agglutinating nature of biomedical language. In this work, we first find that standard open-domain and biomedical tokenizers are largely unable to segment biomedical terms into meaningful components. Therefore, we hypothesize that using a tokenizer which segments biomedical terminology more accurately would enable biomedical LMs to improve their performance on downstream biomedical NLP tasks, especially ones which involve biomedical terms directly such as named entity recognition (NER) and entity linking. Surprisingly, we find that pre-training a biomedical LM using a more accurate biomedical tokenizer does not improve the entity representation quality of a language model as measured by several intrinsic and extrinsic measures such as masked language modeling prediction (MLM) accuracy as well as NER and entity linking performance. These quantitative findings, along with a case study which explores entity representation quality more directly, suggest that the biomedical pre-training process is quite robust to instances of sub-optimal tokenization.

翻译：与普通英语不同，生物医学术语中的许多概念是由生物医学专业人员近期设计的，旨在实现精确性和简洁性。这通常通过组合有意义的生物医学语素来创建新的语义单元实现。然而，大多数现代生物医学语言模型（LM）在预训练时使用的是基于大规模生物医学语料库统计得出的标准领域特定分词器，并未显式利用生物医学语言的粘着性特征。在本研究中，我们首先发现标准开放域和生物医学分词器大多无法将生物医学术语分割为有意义的组成部分。因此，我们假设使用能更准确分割生物医学术语的分词器，将有助于生物医学LM提升在下游生物医学NLP任务中的性能，特别是那些直接涉及生物医学术语的任务（如命名实体识别（NER）和实体链接）。令人惊讶的是，我们发现使用更准确的生物医学分词器预训练生物医学LM，并不能提升语言模型的实体表示质量（通过掩码语言模型预测（MLM）准确率、NER和实体链接性能等多项内在与外在指标衡量）。这些定量结果，结合更直接探索实体表示质量的案例研究，表明生物医学预训练过程对次优分词实例具有相当强的鲁棒性。