Image-text multimodal representation learning aligns data across modalities and enables important medical applications, e.g., image classification, visual grounding, and cross-modal retrieval. In this work, we establish a connection between multimodal representation learning and multiple instance learning. Based on this connection, we propose a generic framework for constructing permutation-invariant score functions with many existing multimodal representation learning approaches as special cases. Furthermore, we use the framework to derive a novel contrastive learning approach and demonstrate that our method achieves state-of-the-art results in several downstream tasks.

翻译：图像-文本多模态表示学习能够对齐跨模态数据，并支持重要的医学应用，例如图像分类、视觉定位和跨模态检索。本研究建立了多模态表示学习与多实例学习之间的联系。基于这一联系，我们提出了一种通用框架，用于构建排列不变的评分函数，其中许多现有的多模态表示学习方法可作为特例。此外，我们利用该框架推导出一种新颖的对比学习方法，并证明该方法在多个下游任务中取得了最先进的结果。