Despite their general capabilities, LLMs still struggle on biomedical NER tasks, which are difficult due to the presence of specialized terminology and lack of training data. In this work we set out to improve LLM performance on biomedical NER in limited data settings via a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly. During this process, to provide a test bed for knowledge augmentation, we perform a comprehensive exploration of prompting strategies. Our experiments show that definition augmentation is useful for both open source and closed LLMs. For example, it leads to a relative improvement of 15\% (on average) in GPT-4 performance (F1) across all (six) of our test datasets. We conduct extensive ablations and analyses to demonstrate that our performance improvements stem from adding relevant definitional knowledge. We find that careful prompting strategies also improve LLM performance, allowing them to outperform fine-tuned language models in few-shot settings. To facilitate future research in this direction, we release our code at https://github.com/allenai/beacon.

翻译：尽管具有通用能力，大语言模型在生物医学命名实体识别任务上仍面临挑战，该任务因专业术语的存在和训练数据匮乏而颇具难度。本研究通过一种新的知识增强方法——在推理过程中实时整合相关概念定义——旨在提升有限数据条件下大语言模型在生物医学命名实体识别上的性能。在实施该方法时，我们通过全面探索提示策略为知识增强提供了测试平台。实验表明，定义增强对开源和闭源大语言模型均有效：例如，在全部六个测试数据集上，该方法使GPT-4的F1值平均相对提升15%。通过广泛的消融实验和分析，我们证实性能提升源于相关定义知识的注入。同时研究发现，精心设计的提示策略也能提升大语言模型性能，使其在少样本场景下超越微调语言模型。为促进该方向的后续研究，我们已在https://github.com/allenai/beacon 公开代码。