The large-scale pre-trained vision language models (VLM) have shown remarkable domain transfer capability on natural images. However, it remains unknown whether this capability can also apply to the medical image domain. This paper thoroughly studies the knowledge transferability of pre-trained VLMs to the medical domain, where we show that well-designed medical prompts are the key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting with expressive attributes that are shared between domains, the VLM can carry the knowledge across domains and improve its generalization. This mechanism empowers VLMs to recognize novel objects with fewer or without image samples. Furthermore, to avoid the laborious manual designing process, we develop three approaches for automatic generation of medical prompts, which can inject expert-level medical knowledge and image-specific information into the prompts for fine-grained grounding. We conduct extensive experiments on thirteen different medical datasets across various modalities, showing that our well-designed prompts greatly improve the zero-shot performance compared to the default prompts, and our fine-tuned models surpass the supervised models by a significant margin.

翻译：大规模预训练的视觉语言模型（VLM）在自然图像上展现出显著的领域迁移能力。然而，这种能力是否同样适用于医学图像领域尚不明确。本文深入研究了预训练VLM向医学领域的知识迁移性，并证明精心设计的医学提示（prompts）是激发预训练VLM知识的关键。我们通过使用领域间共享的表达性属性进行提示，使VLM能够跨领域传递知识并提升其泛化能力。这一机制使得VLM能够以更少甚至无需图像样本的方式识别新对象。此外，为避免繁琐的人工设计过程，我们开发了三种自动生成医学提示的方法，可将专家级医学知识与图像特定信息注入提示中，实现细粒度的语义对齐。我们在涵盖多种模态的十三个不同医学数据集上开展广泛实验，结果表明，与默认提示相比，我们精心设计的提示大幅提升了零样本性能，且微调后的模型显著超越监督式模型。