Foundation vision-language models are currently transforming computer vision, and are on the rise in medical imaging fueled by their very promising generalization capabilities. However, the initial attempts to transfer this new paradigm to medical imaging have shown less impressive performances than those observed in other domains, due to the significant domain shift and the complex, expert domain knowledge inherent to medical-imaging tasks. Motivated by the need for domain-expert foundation models, we present FLAIR, a pre-trained vision-language model for universal retinal fundus image understanding. To this end, we compiled 37 open-access, mostly categorical fundus imaging datasets from various sources, with up to 97 different target conditions and 284,660 images. We integrate the expert's domain knowledge in the form of descriptive textual prompts, during both pre-training and zero-shot inference, enhancing the less-informative categorical supervision of the data. Such a textual expert's knowledge, which we compiled from the relevant clinical literature and community standards, describes the fine-grained features of the pathologies as well as the hierarchies and dependencies between them. We report comprehensive evaluations, which illustrate the benefit of integrating expert knowledge and the strong generalization capabilities of FLAIR under difficult scenarios with domain shifts or unseen categories. When adapted with a lightweight linear probe, FLAIR outperforms fully-trained, dataset-focused models, more so in the few-shot regimes. Interestingly, FLAIR outperforms by a large margin more generalist, larger-scale image-language models, which emphasizes the potential of embedding experts' domain knowledge and the limitations of generalist models in medical imaging.

翻译：基础视觉-语言模型正在颠覆计算机视觉领域，并凭借其强大的泛化能力在医学影像领域兴起。然而，由于显著的领域差异以及医学影像任务固有的复杂专家领域知识，将这一新范式迁移至医学影像的初步尝试所展现的性能不如其他领域亮眼。受领域专家基础模型需求的驱动，我们提出了FLAIR——一种用于通用视网膜眼底图像理解的预训练视觉-语言模型。为此，我们汇编了来自不同来源的37个开放获取、主要为分类性质的眼底成像数据集，涵盖多达97种不同目标病变及284,660张图像。我们在预训练和零样本推理阶段均以描述性文本提示的形式整合专家领域知识，从而增强数据中信息量较少的分类监督信号。这类文本专家知识源自相关临床文献和社区标准，描述了病理的细粒度特征及其层级与依赖关系。我们进行了全面的评估，结果表明整合专家知识的优势以及FLAIR在领域转移或未见类别等困难场景下的强大泛化能力。当使用轻量级线性探针进行适配时，FLAIR性能优于完全训练、以数据集为中心的模型，尤其在少样本场景中更为显著。值得注意的是，FLAIR的性能大幅超越更通用、规模更大的图像-语言模型，这凸显了嵌入专家领域知识的潜力以及通用模型在医学影像领域的局限性。