Zero-shot medical image classification is a critical process in real-world scenarios where we have limited access to all possible diseases or large-scale annotated data. It involves computing similarity scores between a query medical image and possible disease categories to determine the diagnostic result. Recent advances in pretrained vision-language models (VLMs) such as CLIP have shown great performance for zero-shot natural image recognition and exhibit benefits in medical applications. However, an explainable zero-shot medical image recognition framework with promising performance is yet under development. In this paper, we propose a novel CLIP-based zero-shot medical image classification framework supplemented with ChatGPT for explainable diagnosis, mimicking the diagnostic process performed by human experts. The key idea is to query large language models (LLMs) with category names to automatically generate additional cues and knowledge, such as disease symptoms or descriptions other than a single category name, to help provide more accurate and explainable diagnosis in CLIP. We further design specific prompts to enhance the quality of generated texts by ChatGPT that describe visual medical features. Extensive results on one private dataset and four public datasets along with detailed analysis demonstrate the effectiveness and explainability of our training-free zero-shot diagnosis pipeline, corroborating the great potential of VLMs and LLMs for medical applications.

翻译：零样本医学影像分类是真实场景中的关键过程，此时我们难以获取所有可能疾病的样本或大规模标注数据。该方法通过计算查询医学图像与潜在疾病类别之间的相似度得分来确定诊断结果。尽管基于预训练视觉-语言模型（如CLIP）的最新进展在零样本自然图像识别中展现出优异性能，并显示出在医疗领域的应用潜力，但目前尚缺乏兼具可解释性与优异性能的零样本医学影像识别框架。本文提出一种结合ChatGPT的新型CLIP零样本医学影像分类框架，通过模仿人类专家诊断流程实现可解释诊断。核心思想是利用类别名称查询大语言模型，自动生成除单一类别名称之外的额外线索与知识（如疾病症状或描述），从而在CLIP框架中提供更准确且可解释的诊断结果。我们进一步设计了特定提示词以增强ChatGPT生成的描述视觉医学特征文本的质量。基于一个私有数据集与四个公开数据集的广泛实验及详细分析表明，本研究所提免训练零样本诊断流程具备有效性与可解释性，验证了视觉-语言模型与大语言模型在医疗领域的巨大潜力。