In this paper, we consider enhancing medical visual-language pre-training (VLP) with domain-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice. In particular, we make the following contributions: First, unlike existing works that directly process the raw reports, we adopt a novel triplet extraction module to extract the medical-related information, avoiding unnecessary complexity from language grammar and enhancing the supervision signals; Second, we propose a novel triplet encoding module with entity translation by querying a knowledge base, to exploit the rich domain knowledge in medical field, and implicitly build relationships between medical entities in the language embedding space; Third, we propose to use a Transformer-based fusion model for spatially aligning the entity description with visual signals at the image patch level, enabling the ability for medical diagnosis; Fourth, we conduct thorough experiments to validate the effectiveness of our architecture, and benchmark on numerous public benchmarks, e.g., ChestX-ray14, RSNA Pneumonia, SIIM-ACR Pneumothorax, COVIDx CXR-2, COVID Rural, and EdemaSeverity. In both zero-shot and fine-tuning settings, our model has demonstrated strong performance compared with the former methods on disease classification and grounding.

翻译：在本文中，我们考虑通过利用放射科日常实践中的配对图像-文本报告，将领域特定知识融入医学视觉-语言预训练（VLP）中。具体而言，我们做出以下贡献：第一，不同于现有直接处理原始报告的工作，我们采用新颖的三元组提取模块提取医学相关信息，避免了语言语法带来的不必要复杂性，并增强了监督信号；第二，我们提出一种新颖的三元组编码模块，通过查询知识库进行实体翻译，以挖掘医学领域丰富的领域知识，并在语言嵌入空间中隐式构建医学实体间的关系；第三，我们提出使用基于Transformer的融合模型，在图像块级别上对实体描述与视觉信号进行空间对齐，从而具备医学诊断能力；第四，我们开展了充分的实验验证了架构的有效性，并在多个公开基准（如ChestX-ray14、RSNA肺炎、SIIM-ACR气胸、COVIDx CXR-2、COVID Rural及EdemaSeverity）上进行评估。在零样本和微调两种设定下，我们的模型在疾病分类和定位任务上均展现出优于以往方法的强劲性能。