In this paper, we present HuatuoGPT, a large language model (LLM) for medical consultation. The core recipe of HuatuoGPT is to leverage both \textit{distilled data from ChatGPT} and \textit{real-world data from doctors} in the supervised fine-tuned stage. The responses of ChatGPT are usually detailed, well-presented and informative while it cannot perform like a doctor in many aspects, e.g. for integrative diagnosis. We argue that real-world data from doctors would be complementary to distilled data in the sense the former could tame a distilled language model to perform like doctors. To better leverage the strengths of both data, we train a reward model to align the language model with the merits that both data bring, following an RLAIF (reinforced learning from AI feedback) fashion. To evaluate and benchmark the models, we propose a comprehensive evaluation scheme (including automatic and manual metrics). Experimental results demonstrate that HuatuoGPT achieves state-of-the-art results in performing medical consultation among open-source LLMs in GPT-4 evaluation, human evaluation, and medical benchmark datasets. It is worth noting that by using additional real-world data and RLAIF, the distilled language model (i.e., HuatuoGPT) outperforms its teacher model ChatGPT in most cases. Our code, data, and models are publicly available at \url{https://github.com/FreedomIntelligence/HuatuoGPT}. The online demo is available at \url{https://www.HuatuoGPT.cn/}.

翻译：本文提出HuatuoGPT，一种用于医学咨询的大型语言模型（LLM）。其核心方法在于监督微调阶段同时利用来自ChatGPT的蒸馏数据和来自医生的真实世界数据。ChatGPT的回答通常详尽、表述清晰且信息丰富，但在综合诊断等多方面无法像医生一样工作。我们认为，真实的医生数据能够补充蒸馏数据，因为前者可以驯服蒸馏语言模型使其表现得更像医生。为更好地发挥两类数据的优势，我们遵循RLAIF（基于AI反馈的强化学习）范式，训练一个奖励模型使语言模型与两类数据带来的优点对齐。为评估和基准化模型，我们提出一套全面的评估方案（包括自动和人工指标）。实验结果表明，在GPT-4评估、人工评估和医学基准数据集上，HuatuoGPT在开源LLM中取得了医学咨询的最佳结果。值得注意的是，通过使用额外的真实世界数据和RLAIF，蒸馏语言模型（即HuatuoGPT）在多数情况下优于其教师模型ChatGPT。我们的代码、数据和模型已在https://github.com/FreedomIntelligence/HuatuoGPT公开，在线演示可在https://www.HuatuoGPT.cn/获取。