Large language models (LLMs), such as GPT-4, have demonstrated remarkable capabilities across a wide range of tasks, including health applications. In this paper, we study how LLMs can be used to scale biomedical knowledge curation. We find that while LLMs already possess decent competency in structuring biomedical text, by distillation into a task-specific student model through self-supervised learning, substantial gains can be attained over out-of-box LLMs, with additional advantages such as cost, efficiency, and white-box model access. We conduct a case study on adverse drug event (ADE) extraction, which is an important area for improving care. On standard ADE extraction evaluation, a GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised state-of-the-art models without using any labeled data. Despite being over 1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by over 6 absolute points in F1 and GPT-4 by over 5 absolute points. Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT) and ADE extraction architecture shed light on best practice for biomedical knowledge extraction. Similar gains were attained by distillation for other standard biomedical knowledge extraction tasks such as gene-disease associations and protected health information, further illustrating the promise of this approach.

翻译：大型语言模型（LLMs），如GPT-4，在包括健康应用在内的广泛任务中展现了卓越能力。本文研究如何利用LLMs规模化生物医学知识整理。我们发现，尽管LLMs在结构化生物医学文本方面已具备相当能力，但通过自监督学习将其蒸馏为任务专用学生模型，可获得超越开箱即用LLMs的显著增益，并具备成本、效率及白盒模型访问等额外优势。我们以药物不良事件（ADE）抽取作为案例研究，该领域对改善医疗至关重要。在标准ADE抽取评估中，GPT-3.5蒸馏得到的PubMedBERT模型无需使用任何标注数据，即达到与监督式最先进模型相当的准确率。尽管模型规模缩小超过1000倍，蒸馏模型在F1指标上仍比其教师模型GPT-3.5高出6个绝对百分点，比GPT-4高出5个绝对百分点。关于蒸馏模型选择（如PubMedBERT对比BioGPT）及ADE抽取架构的消融研究，揭示了生物医学知识抽取的最佳实践。将蒸馏方法应用于其他标准生物医学知识抽取任务（如基因-疾病关联及受保护健康信息）同样获得类似增益，进一步验证了该方法的潜力。