Multi-modal reasoning plays a vital role in bridging the gap between textual and visual information, enabling a deeper understanding of the context. This paper presents the Feature Swapping Multi-modal Reasoning (FSMR) model, designed to enhance multi-modal reasoning through feature swapping. FSMR leverages a pre-trained visual-language model as an encoder, accommodating both text and image inputs for effective feature representation from both modalities. It introduces a unique feature swapping module, enabling the exchange of features between identified objects in images and corresponding vocabulary words in text, thereby enhancing the model's comprehension of the interplay between images and text. To further bolster its multi-modal alignment capabilities, FSMR incorporates a multi-modal cross-attention mechanism, facilitating the joint modeling of textual and visual information. During training, we employ image-text matching and cross-entropy losses to ensure semantic consistency between visual and language elements. Extensive experiments on the PMR dataset demonstrate FSMR's superiority over state-of-the-art baseline models across various performance metrics.

翻译：多模态推理在弥合文本与视觉信息之间的鸿沟、实现上下文深层理解方面发挥着关键作用。本文提出特征交换多模态推理（FSMR）模型，通过特征交换机制增强多模态推理能力。FSMR采用预训练的视觉语言模型作为编码器，同时处理文本与图像输入，实现两种模态的有效特征表征。该模型引入独特的特征交换模块，可在图像中识别物体与文本中对应词汇之间进行特征交换，从而增强模型对图像与文本交互关系的理解。为进一步提升多模态对齐能力，FSMR融入多模态交叉注意力机制，实现文本与视觉信息的联合建模。训练过程中，我们采用图像-文本匹配损失与交叉熵损失，确保视觉与语言元素间的语义一致性。在PMR数据集上的大量实验表明，FSMR在多项性能指标上均优于现有最优基线模型。