Recent advances in natural language processing (NLP) owe their success to pre-training language models on large amounts of unstructured data. Still, there is an increasing effort to combine the unstructured nature of LMs with structured knowledge and reasoning. Particularly in the rapidly evolving field of biomedical NLP, knowledge-enhanced language models (KELMs) have emerged as promising tools to bridge the gap between large language models and domain-specific knowledge, considering the available biomedical knowledge graphs (KGs) curated by experts over the decades. In this paper, we develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models (PLMs). We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical ontology OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. The approach includes partitioning knowledge graphs into smaller subgraphs, fine-tuning adapter modules for each subgraph, and combining the knowledge in a fusion layer. We test the performance on three downstream tasks: document classification,question answering, and natural language inference. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low. Finally, we provide a detailed interpretation of the results and report valuable insights for future work.

翻译：自然语言处理（NLP）的最新进展得益于在大规模非结构化数据上预训练语言模型。然而，研究者日益致力于将语言模型的非结构化特性与结构化知识及推理能力相结合。特别是在快速发展的生物医学NLP领域，知识增强型语言模型（KELMs）已成为连接大型语言模型与领域特定知识的有前景工具，其基础是数十年来由专家精心整理的生物医学知识图谱（KGs）。本文提出了一种方法，利用轻量级适配器模块将结构化生物医学知识注入预训练语言模型（PLMs）。我们采用两个大型知识图谱——生物医学知识系统UMLS与新型生物化学本体OntoChem，并配合两个著名的生物医学PLM模型——PubMedBERT与BioLinkBERT。该方法包括将知识图谱划分为更小的子图、为每个子图微调适配器模块，以及在融合层中整合知识。我们在三个下游任务上测试了性能：文档分类、问答推理与自然语言推断。实验表明，我们的方法在多个实例中提升了性能，同时保持了较低的计算资源需求。最后，我们对结果进行了详细解读，并为未来研究提供了有价值的见解。