Multimodal large language models (LLMs) are increasingly used to generate dermatology diagnostic narratives directly from images. However, reliable evaluation remains the primary bottleneck for responsible clinical deployment. We introduce a novel evaluation framework that combines DermBench, a meticulously curated benchmark, with DermEval, a robust automatic evaluator, to enable clinically meaningful, reproducible, and scalable assessment. We build DermBench, which pairs 4,000 real-world dermatology images with expert-certified diagnostic narratives and uses an LLM-based judge to score candidate narratives across clinically grounded dimensions, enabling consistent and comprehensive evaluation of multimodal models. For individual case assessment, we train DermEval, a reference-free multimodal evaluator. Given an image and a generated narrative, DermEval produces a structured critique along with an overall score and per-dimension ratings. This capability enables fine-grained, per-case analysis, which is critical for identifying model limitations and biases. Experiments on a diverse dataset of 4,500 cases demonstrate that DermBench and DermEval achieve close alignment with expert ratings, with mean deviations of 0.251 and 0.117 (out of 5), respectively, providing reliable measurement of diagnostic ability and trustworthiness across different multimodal LLMs.

翻译：多模态大语言模型正被日益用于直接根据图像生成皮肤病学诊断叙述。然而，可靠的评估仍然是实现负责任临床部署的主要瓶颈。我们提出了一种新颖的评估框架，该框架将精心构建的基准DermBench与鲁棒的自动评估器DermEval相结合，以实现具有临床意义、可复现且可扩展的评估。我们构建了DermBench，它包含了4,000张真实世界皮肤病学图像与经过专家认证的诊断叙述配对，并利用基于大语言模型的评判器，在基于临床的多个维度上对候选叙述进行评分，从而实现对多模态模型一致且全面的评估。针对单个病例的评估，我们训练了无需参考的多模态评估器DermEval。给定一张图像和一段生成的叙述，DermEval能够生成一份结构化评析，同时提供一个总体分数以及各维度评分。此功能支持细粒度的逐病例分析，这对于识别模型的局限性和偏差至关重要。在包含4,500个病例的多样化数据集上的实验表明，DermBench和DermEval与专家评分高度吻合，平均偏差分别为0.251和0.117（满分5分），从而为不同多模态大语言模型的诊断能力和可信度提供了可靠的度量。