Medical image classification is a critical problem for healthcare, with the potential to alleviate the workload of doctors and facilitate diagnoses of patients. However, two challenges arise when deploying deep learning models to real-world healthcare applications. First, neural models tend to learn spurious correlations instead of desired features, which could fall short when generalizing to new domains (e.g., patients with different ages). Second, these black-box models lack interpretability. When making diagnostic predictions, it is important to understand why a model makes a decision for trustworthy and safety considerations. In this paper, to address these two limitations, we propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model. We systematically evaluate our method on eight medical image classification datasets to verify its effectiveness. On challenging datasets with strong confounding factors, our method can mitigate spurious correlations thus substantially outperform standard visual encoders and other baselines. Finally, we show how classification with a small number of concepts brings a level of interpretability for understanding model decisions through case studies in real medical data.

翻译：医学图像分类是医疗健康领域的关键问题，有望减轻医生工作负担并辅助患者诊断。然而，将深度学习模型部署到实际医疗应用时面临两个挑战：首先，神经模型倾向于学习伪相关而非期望特征，这在泛化到新领域（如不同年龄患者群体）时可能表现不佳；其次，这类黑箱模型缺乏可解释性。在进行诊断预测时，出于可信度和安全性考量，理解模型决策依据至关重要。为应对这两大局限，本文提出一种基于自然语言概念构建鲁棒且可解释医学图像分类器的新范式。具体而言，我们首先从GPT-4中查询临床概念，随后通过视觉语言模型将潜在图像特征转化为显式概念。我们在八个医学图像分类数据集上系统评估了该方法，验证其有效性。在存在强混杂因素的挑战性数据集中，本方法能有效缓解伪相关，从而显著优于标准视觉编码器及其他基线方法。最后，通过真实医疗数据案例研究，我们展示了基于少量概念的分类如何为理解模型决策带来可解释性。