An important component of human analysis of medical images and their context is the ability to relate newly seen things to related instances in our memory. In this paper we mimic this ability by using multi-modal retrieval augmentation and apply it to several tasks in chest X-ray analysis. By retrieving similar images and/or radiology reports we expand and regularize the case at hand with additional knowledge, while maintaining factual knowledge consistency. The method consists of two components. First, vision and language modalities are aligned using a pre-trained CLIP model. To enforce that the retrieval focus will be on detailed disease-related content instead of global visual appearance it is fine-tuned using disease class information. Subsequently, we construct a non-parametric retrieval index, which reaches state-of-the-art retrieval levels. We use this index in our downstream tasks to augment image representations through multi-head attention for disease classification and report retrieval. We show that retrieval augmentation gives considerable improvements on these tasks. Our downstream report retrieval even shows to be competitive with dedicated report generation methods, paving the path for this method in medical imaging.

翻译：人类分析医学影像及其背景的重要组成部分，是将新观察到的内容与记忆中相关实例建立联系的能力。本文通过多模态检索增强模拟该能力，并将其应用于胸部X光分析的多个任务。通过检索相似影像和/或放射学报告，我们在保持事实知识一致性的同时，利用额外知识扩展并规范当前病例。该方法包含两个组成部分：首先，使用预训练的CLIP模型对齐视觉与语言模态。为强制检索聚焦于细粒度疾病相关内容而非全局视觉外观，我们利用疾病类别信息对其进行微调。随后构建非参数检索索引，达到当前最优检索水平。在下游任务中，我们通过多头注意力机制利用该索引增强影像表征，用于疾病分类与报告检索。实验表明，检索增强显著提升了这些任务的性能。其中下游报告检索任务甚至展现出与专用报告生成方法相匹敌的能力，为该方法在医学影像领域的应用开辟了新路径。