Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.

翻译：医疗对话人工智能在开发更安全、更有效的医疗对话系统中发挥着关键作用。然而，现有用于评估医疗大语言模型信息收集与诊断推理能力的基准和评估框架尚未经过严格验证。为弥补这些不足，我们提出了MedDialogRubrics——一个包含5,200个合成构建的病例和60,000余条细粒度评估准则的新型基准，这些准则由大语言模型生成并经临床专家优化，专门用于评估大语言模型的多轮诊断能力。我们的框架采用多智能体系统，基于疾病知识合成真实的患者病历和主诉，无需访问真实世界电子健康记录，从而规避隐私和数据治理风险。我们设计了鲁棒的患者智能体，其仅能访问原子化医学事实集，并通过动态引导机制持续检测并修正对话中的幻觉，确保模拟病例的内在一致性与临床合理性。此外，我们提出了基于大语言模型的结构化准则生成流程，该流程检索循证医学指南并利用拒绝采样技术，为每个病例生成优先级准则项（"必问"项目）。我们对前沿模型进行了全面评估，结果表明当前模型在多个评估维度上面临重大挑战。研究结果指出，提升医疗对话能力需要对话管理架构的革新，而不仅是对基础模型的渐进式调优。