As the immense opportunities enabled by large language models become more apparent, NLP systems will be increasingly expected to excel in real-world settings. However, in many instances, powerful models alone will not yield translational NLP solutions, especially if the formulated problem is not well aligned with the real-world task. In this work, we study the case of UMLS vocabulary insertion, an important real-world task in which hundreds of thousands of new terms, referred to as atoms, are added to the UMLS, one of the most comprehensive open-source biomedical knowledge bases. Previous work aimed to develop an automated NLP system to make this time-consuming, costly, and error-prone task more efficient. Nevertheless, practical progress in this direction has been difficult to achieve due to a problem formulation and evaluation gap between research output and the real-world task. In order to address this gap, we introduce a new formulation for UMLS vocabulary insertion which mirrors the real-world task, datasets which faithfully represent it and several strong baselines we developed through re-purposing existing solutions. Additionally, we propose an effective rule-enhanced biomedical language model which enables important new model behavior, outperforms all strong baselines and provides measurable qualitative improvements to editors who carry out the UVI task. We hope this case study provides insight into the considerable importance of problem formulation for the success of translational NLP solutions.

翻译：随着大语言模型所赋能的巨大机遇日益显现，自然语言处理系统有望在真实场景中表现出色。然而在许多情况下，仅凭强大的模型本身无法带来转化型NLP解决方案，尤其是当所构建的问题与真实任务未能很好对齐时。本研究聚焦于UMLS词汇插入这一重要真实任务——该任务需将数十万计的新术语（称为原子）添加至UMLS（最全面的开源生物医学知识库之一）。先前研究致力于开发自动化NLP系统，以提升这一耗时、昂贵且易错任务的效率。然而，由于研究产出与真实任务之间存在问题构建和评估的差距，该方向的实际进展一直困难重重。为弥合这一差距，我们提出了与真实任务镜像对应的UMLS词汇插入新问题构建方案、忠实反映该任务的数据集，以及通过重新利用现有方案开发的多个强基线模型。此外，我们提出了一种有效的规则增强型生物医学语言模型，该模型实现了重要的新型模型行为，在性能上优于所有强基线模型，并为执行UVI任务的编辑人员提供了可量化的质量提升。我们希望本案例研究能揭示问题构建对于转化型NLP解决方案成功的关键作用。