CLIP and BiomedCLIP are examples of vision-language foundation models and offer strong cross-modal embeddings; however, they are not optimized for fine-grained medical retrieval tasks, such as retrieving clinically relevant radiology reports using chest X-ray (CXR) image queries. To address this shortcoming, we propose a multi-task learning framework to fine-tune BiomedCLIP and evaluate improvements to CXR image-text retrieval. Using BiomedCLIP as the backbone, we incorporate a lightweight MLP projector head trained with a multi-task composite loss function that includes: (1) a binary cross-entropy loss to distinguish normal from abnormal CXR studies, (2) a supervised contrastive loss to reinforce intra-class consistency, and (3) a CLIP loss to maintain cross-modal alignment. Experimental results demonstrate that the fine-tuned model achieves more balanced and clinically meaningful performance across both image-to-text and text-to-image retrieval tasks compared to the pretrained BiomedCLIP and general-purpose CLIP models. Furthermore, t-SNE visualizations reveal clearer semantic clustering of normal and abnormal cases, demonstrating the model's enhanced diagnostic sensitivity. These findings highlight the value of domain-adaptive, multi-task learning for advancing cross-modal retrieval in biomedical applications.

翻译：CLIP和BiomedCLIP是视觉-语言基础模型的代表，提供了强大的跨模态嵌入能力；然而，它们并未针对细粒度医学检索任务（例如使用胸部X光图像查询检索临床相关的放射学报告）进行优化。为弥补这一不足，我们提出一种多任务学习框架来微调BiomedCLIP，并评估其在胸部X光图像-文本检索任务上的改进效果。以BiomedCLIP为骨干网络，我们引入一个轻量级MLP投影头，该投影头通过包含以下三部分的多任务复合损失函数进行训练：（1）用于区分正常与异常胸部X光研究的二元交叉熵损失；（2）用于增强类内一致性的监督对比损失；（3）用于保持跨模态对齐的CLIP损失。实验结果表明，与预训练的BiomedCLIP及通用CLIP模型相比，微调后的模型在图像到文本和文本到图像检索任务上均取得了更均衡且更具临床意义的性能。此外，t-SNE可视化结果显示正常与异常病例的语义聚类更为清晰，证明了模型增强的诊断敏感性。这些发现凸显了领域自适应的多任务学习在推动生物医学跨模态检索应用中的价值。