Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in various multimodal tasks. However, their potential in the medical domain remains largely unexplored. A significant challenge arises from the scarcity of diverse medical images spanning various modalities and anatomical regions, which is essential in real-world medical applications. To solve this problem, in this paper, we introduce OmniMedVQA, a novel comprehensive medical Visual Question Answering (VQA) benchmark. This benchmark is collected from 75 different medical datasets, including 12 different modalities and covering more than 20 distinct anatomical regions. Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs. Through our extensive experiments, we have found that existing LVLMs struggle to address these medical VQA problems effectively. Moreover, what surprises us is that medical-specialized LVLMs even exhibit inferior performance to those general-domain models, calling for a more versatile and robust LVLM in the biomedical field. The evaluation results not only reveal the current limitations of LVLM in understanding real medical images but also highlight our dataset's significance. Our dataset will be made publicly available.

翻译：大型视觉语言模型（LVLMs）已在多种多模态任务中展现出卓越能力，但其在医学领域的潜力仍有待深入探索。一个关键挑战在于缺乏涵盖多种模态和解剖区域的多样化医学图像，而这在真实医学应用中至关重要。为解决此问题，本文提出OmniMedVQA，一个全新的医学视觉问答（VQA）综合基准。该基准汇集了75个不同的医学数据集，涵盖12种不同模态及超过20个不同解剖区域。重要的是，基准中所有图像均源自真实医学场景，确保了与医学领域需求的契合性及评估LVLMs的适用性。通过大量实验，我们发现现有LVLMs难以有效解决这些医学VQA问题。更令人惊讶的是，医学专用LVLMs的表现甚至逊于通用领域模型，这表明生物医学领域亟需更通用、更稳健的LVLM。评估结果不仅揭示了当前LVLM在理解真实医学图像方面的局限性，也凸显了我们数据集的重要性。本数据集将公开发布。