Large language models (LLMs) have demonstrated powerful text generation capabilities, bringing unprecedented innovation to the healthcare field. While LLMs hold immense promise for applications in healthcare, applying them to real clinical scenarios presents significant challenges, as these models may generate content that deviates from established medical facts and even exhibit potential biases. In our research, we develop an augmented LLM framework based on the Unified Medical Language System (UMLS), aiming to better serve the healthcare community. We employ LLaMa2-13b-chat and ChatGPT-3.5 as our benchmark models, and conduct automatic evaluations using the ROUGE Score and BERTScore on 104 questions from the LiveQA test set. Additionally, we establish criteria for physician-evaluation based on four dimensions: Factuality, Completeness, Readability and Relevancy. ChatGPT-3.5 is used for physician evaluation with 20 questions on the LiveQA test set. Multiple resident physicians conducted blind reviews to evaluate the generated content, and the results indicate that this framework effectively enhances the factuality, completeness, and relevance of generated content. Our research demonstrates the effectiveness of using UMLS-augmented LLMs and highlights the potential application value of LLMs in in medical question-answering.

翻译：大型语言模型（LLMs）已展现出强大的文本生成能力，为医疗领域带来了前所未有的创新。尽管LLMs在医疗领域的应用前景广阔，但将其应用于真实临床场景仍面临重大挑战，因为这些模型可能生成偏离既定医学事实甚至存在潜在偏见的内容。在本研究中，我们开发了一种基于统一医学语言系统（UMLS）的增强型LLM框架，旨在更好地服务于医疗社区。我们采用LLaMa2-13b-chat和ChatGPT-3.5作为基准模型，并在LiveQA测试集的104个问题上使用ROUGE分数和BERTScore进行自动评估。此外，我们基于事实性、完整性、可读性和相关性四个维度建立了医生评估标准。使用ChatGPT-3.5对LiveQA测试集的20个问题进行医生评估，多名住院医师通过盲审对生成内容进行评价。结果表明，该框架有效提升了生成内容的事实性、完整性和相关性。我们的研究验证了使用UMLS增强LLMs的有效性，并突显了LLMs在医疗问答中的潜在应用价值。