Large Language Models (LLMs) have significantly advanced healthcare innovation on generation capabilities. However, their application in real clinical settings is challenging due to potential deviations from medical facts and inherent biases. In this work, we develop an augmented LLM framework, KG-Rank, which leverages a medical knowledge graph (KG) with ranking and re-ranking techniques, aiming to improve free-text question-answering (QA) in the medical domain. Specifically, upon receiving a question, we initially retrieve triplets from a medical KG to gather factual information. Subsequently, we innovatively apply ranking methods to refine the ordering of these triplets, aiming to yield more precise answers. To the best of our knowledge, KG-Rank is the first application of ranking models combined with KG in medical QA specifically for generating long answers. Evaluation of four selected medical QA datasets shows that KG-Rank achieves an improvement of over 18% in the ROUGE-L score. Moreover, we extend KG-Rank to open domains, where it realizes a 14% improvement in ROUGE-L, showing the effectiveness and potential of KG-Rank.

翻译：摘要：大语言模型凭借其生成能力显著推动了医疗领域的创新。然而，由于可能偏离医学事实并存在固有偏差，其在真实临床场景中的应用仍面临挑战。本研究开发了一种增强型大语言模型框架KG-Rank，该框架通过结合医学知识图谱与排序及重排序技术，旨在提升医学领域的自由文本问答性能。具体而言，当接收到问题时，我们首先从医学知识图谱中检索三元组以获取事实性信息，随后创新性地应用排序方法优化这些三元组的排列顺序，从而生成更精准的答案。据我们所知，KG-Rank是首个将排序模型与知识图谱相结合以生成长篇答案的医学问答应用。在四个选定的医学问答数据集上的评估表明，KG-Rank在ROUGE-L指标上实现了超过18%的提升。此外，我们将KG-Rank扩展至开放域场景，其ROUGE-L值提升14%，验证了该框架的有效性与潜力。