Medical imaging analysis plays a critical role in the diagnosis and treatment of various medical conditions. This paper focuses on chest X-ray images and their corresponding radiological reports. It presents a new model that learns a joint X-ray image & report representation. The model is based on a novel alignment scheme between the visual data and the text, which takes into account both local and global information. Furthermore, the model integrates domain-specific information of two types -- lateral images and the consistent visual structure of chest images. Our representation is shown to benefit three types of retrieval tasks: text-image retrieval, class-based retrieval, and phrase-grounding.

翻译：医学影像分析在各种疾病的诊断和治疗中扮演着关键角色。本文聚焦于胸部X光图像及其对应的放射学报告，提出了一种学习联合X光图像与报告表示的新模型。该模型基于视觉数据与文本之间新颖的对齐方案，同时考虑了局部与全局信息。此外，模型整合了两类领域特定信息——侧位图像与胸部图像一致的视觉结构。实验表明，我们的表示方法对三类检索任务均有裨益：文本-图像检索、基于类别的检索以及短语定位。