Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.

翻译：医疗对话人工智能（AI）在开发更安全、更有效的医疗对话系统中发挥着关键作用。然而，现有用于评估医疗大语言模型（LLMs）信息收集与诊断推理能力的基准和评估框架尚未经过严格评估。为填补这些空白，我们提出了MedDialogRubrics——一个包含5,200个合成构建的患者案例及超过60,000条由LLMs生成并经临床专家细化的细粒度评估量规的新型基准，专门用于评估LLMs的多轮诊断能力。我们的框架采用多智能体系统，基于底层疾病知识合成真实的患者病历和主诉，而无需访问真实世界的电子健康记录，从而规避了隐私和数据治理问题。我们设计了一个稳健的患者智能体，其仅能访问一组原子医疗事实，并配备了动态引导机制，可在整个对话过程中持续检测并纠正幻觉，确保模拟案例的内部一致性与临床合理性。此外，我们提出了一种基于LLM的结构化且经专家标注的量规生成流程，该流程检索循证医学（EBM）指南，并利用拒绝采样为每个案例推导出优先级的量规条目（“必问”条目）。我们对前沿模型进行了全面评估，结果表明在多个评估维度上，当前模型均面临重大挑战。我们的研究结果指出，改进医疗对话需要对话管理架构的进步，而不仅仅是基础模型的增量调优。