Evaluating large language models (LLMs) has recently emerged as a critical issue for safe and trustworthy application of LLMs in the medical domain. Although a variety of static medical question-answering (QA) benchmarks have been proposed, many aspects remain underexplored, such as the effectiveness of LLMs in generating responses in dynamic, interactive clinical multi-turn conversation situations and the identification of multi-faceted evaluation strategies beyond simple accuracy. However, formally evaluating a dynamic, interactive clinical situation is hindered by its vast combinatorial space of possible patient states and interaction trajectories, making it difficult to standardize and quantitatively measure such scenarios. Here, we introduce AutoMedic, a multi-agent simulation framework that enables automated evaluation of LLMs as clinical conversational agents. AutoMedic transforms off-the-shelf static QA datasets into virtual patient profiles, enabling realistic and clinically grounded multi-turn clinical dialogues between LLM agents. The performance of various clinical conversational agents is then assessed based on our CARE metric, which provides a multi-faceted evaluation standard of clinical conversational accuracy, efficiency/strategy, empathy, and robustness. Our findings, validated by human experts, demonstrate the validity of AutoMedic as an automated evaluation framework for clinical conversational agents, offering practical guidelines for the effective development of LLMs in conversational medical applications.

翻译：大型语言模型（LLMs）的评估已成为其在医疗领域安全可信应用的关键问题。尽管已有多种静态医学问答（QA）基准被提出，但许多方面仍待深入探索，例如LLMs在动态、交互式临床多轮对话情境中生成响应的有效性，以及超越简单准确性的多维度评估策略的识别。然而，正式评估动态交互式临床情境受限于其庞大的患者状态与交互轨迹组合空间，难以对此类场景进行标准化和定量测量。为此，我们提出AutoMedic，一种多智能体仿真框架，能够自动化评估作为临床对话智能体的LLMs。AutoMedic将现成的静态QA数据集转化为虚拟患者档案，支持LLM智能体之间进行基于临床实际的多轮临床对话。随后，各类临床对话智能体的性能通过我们的CARE指标进行评估，该指标提供了临床对话准确性、效率/策略、共情能力及鲁棒性的多维度评估标准。经人类专家验证的研究结果证明了AutoMedic作为临床对话智能体自动化评估框架的有效性，为LLMs在对话式医疗应用中的高效开发提供了实用指导。