Medical image representations can be learned through medical vision-language contrastive learning (mVLCL) where medical imaging reports are used as weak supervision through image-text alignment. These learned image representations can be transferred to and benefit various downstream medical vision tasks such as disease classification and segmentation. Recent mVLCL methods attempt to align image sub-regions and the report keywords as local-matchings. However, these methods aggregate all local-matchings via simple pooling operations while ignoring the inherent relations between them. These methods therefore fail to reason between local-matchings that are semantically related, e.g., local-matchings that correspond to the disease word and the location word (semantic-relations), and also fail to differentiate such clinically important local-matchings from others that correspond to less meaningful words, e.g., conjunction words (importance-relations). Hence, we propose a mVLCL method that models the inter-matching relations between local-matchings via a relation-enhanced contrastive learning framework (RECLF). In RECLF, we introduce a semantic-relation reasoning module (SRM) and an importance-relation reasoning module (IRM) to enable more fine-grained report supervision for image representation learning. We evaluated our method using six public benchmark datasets on four downstream tasks, including segmentation, zero-shot classification, linear classification, and cross-modal retrieval. Our results demonstrated the superiority of our RECLF over the state-of-the-art mVLCL methods with consistent improvements across single-modal and cross-modal tasks. These results suggest that our RECLF, by modelling the inter-matching relations, can learn improved medical image representations with better generalization capabilities.

翻译：医学图像表示可通过医学视觉-语言对比学习（mVLCL）进行学习，其中医学影像报告通过图像-文本对齐作为弱监督信号。这些学习到的图像表示可迁移至多种下游医学视觉任务（如疾病分类与分割）并提升其性能。现有mVLCL方法尝试将图像子区域与报告关键词作为局部匹配进行对齐，但仅通过简单池化操作聚合所有局部匹配，忽略了其内在关联关系。此类方法因而无法在语义相关的局部匹配间进行推理（例如对应疾病词汇与位置词汇的局部匹配间的语义关联），亦无法区分此类临床重要的局部匹配与对应非关键词汇（如连接词）的局部匹配之间的重要性差异。为此，我们提出一种通过关系增强对比学习框架（RECLF）建模局部匹配间跨匹配关系的mVLCL方法。在RECLF中，我们引入语义关系推理模块（SRM）与重要性关系推理模块（IRM），为图像表示学习提供更细粒度的报告监督。我们在六个公开基准数据集上通过分割、零样本分类、线性分类及跨模态检索四项下游任务评估所提方法。实验结果表明，我们的RECLF在单模态与跨模态任务中均取得一致性能提升，显著优于当前最先进的mVLCL方法。这些结果证明，通过建模跨匹配关系，我们的RECLF能够学习具有更优泛化能力的医学图像表示。