Defining pathologies automatically from medical images aids the understanding of the emergence and progression of diseases, and such an ability is crucial in clinical diagnostics. However, existing deep learning models heavily rely on expert annotations and lack generalization capabilities in open clinical environments. In this study, we present a generalizable vision-language pre-training model for Annotation-Free pathological lesions Localization (AFLoc). The core strength of AFLoc lies in its extensive multi-level semantic structure-based contrastive learning, which comprehensively aligns multi-granularity medical concepts from reports with abundant image features, to adapt to the diverse expressions of pathologies and unseen pathologies without the reliance on image annotations from experts. We demonstrate the proof of concept on CXR images, with extensive experimental validation across 4 distinct external datasets, encompassing 11 types of chest pathologies. The results demonstrate that AFLoc surpasses state-of-the-art methods in pathological lesions localization and disease classification, and even outperforms the human benchmark in locating 5 different pathologies. Additionally, we further verify its generalization ability by applying it to retinal fundus images. Our approach showcases AFoc versatilities and underscores its suitability for clinical diagnoses in complex clinical environments.

翻译：从医学图像中自动定义病理特征有助于理解疾病的发生与发展，这种能力在临床诊断中至关重要。然而，现有深度学习模型严重依赖专家标注，且在开放式临床环境中缺乏泛化能力。本研究提出一种具有泛化能力的视觉语言预训练模型AFLoc（无标注病理病灶定位）。AFLoc的核心优势在于其基于多层次语义结构的对比学习机制，能够将报告中的多粒度医学概念与丰富的图像特征进行全局对齐，从而适应病理特征的多样化表述及未见过的病理类型，且无需依赖专家图像标注。我们在胸部X光图像上验证了这一概念，并在涵盖11种胸部病理类型的4个不同外部数据集上进行了广泛实验。结果表明，AFLoc在病理病灶定位和疾病分类任务上均优于现有最优方法，甚至在5种不同病理的定位中超越了人类基准。此外，我们通过将其应用于视网膜眼底图像进一步验证了其泛化能力。本方法展现了AFLoc的多功能性，并突显其适用于复杂临床环境中的临床诊断。