We study the problem of multimodal fusion in this paper. Recent exchanging-based methods have been proposed for vision-vision fusion, which aim to exchange embeddings learned from one modality to the other. However, most of them project inputs of multimodalities into different low-dimensional spaces and cannot be applied to the sequential input data. To solve these issues, in this paper, we propose a novel exchanging-based multimodal fusion model MuSE for text-vision fusion based on Transformer. We first use two encoders to separately map multimodal inputs into different low-dimensional spaces. Then we employ two decoders to regularize the embeddings and pull them into the same space. The two decoders capture the correlations between texts and images with the image captioning task and the text-to-image generation task, respectively. Further, based on the regularized embeddings, we present CrossTransformer, which uses two Transformer encoders with shared parameters as the backbone model to exchange knowledge between multimodalities. Specifically, CrossTransformer first learns the global contextual information of the inputs in the shallow layers. After that, it performs inter-modal exchange by selecting a proportion of tokens in one modality and replacing their embeddings with the average of embeddings in the other modality. We conduct extensive experiments to evaluate the performance of MuSE on the Multimodal Named Entity Recognition task and the Multimodal Sentiment Analysis task. Our results show the superiority of MuSE against other competitors. Our code and data are provided at https://github.com/RecklessRonan/MuSE.

翻译：本文研究多模态融合问题。近期提出了用于视觉-视觉融合的基于交换机制的方法，旨在将一个模态学习到的嵌入表示交换至另一模态。然而，大多数方法将多模态输入投影到不同的低维空间，无法适用于序列输入数据。为解决这些问题，本文提出一种新颖的基于交换机制的文本-视觉多模态融合模型MuSE，该模型基于Transformer架构。我们首先使用两个编码器将多模态输入分别映射到不同的低维空间，随后采用两个解码器对嵌入表示进行规范化处理，并将其拉入同一空间。这两个解码器分别通过图像描述任务和文本到图像生成任务捕获文本与图像之间的相关性。进一步地，基于规范化后的嵌入表示，我们提出CrossTransformer，采用两个参数共享的Transformer编码器作为主干模型，实现多模态间的知识交换。具体而言，CrossTransformer先在浅层学习输入的全局上下文信息，然后通过选取某一模态中的部分标记，并将其嵌入替换为另一模态的嵌入平均值，执行模态间交换。我们针对多模态命名实体识别任务和多模态情感分析任务进行了大量实验以评估MuSE的性能。结果表明，MuSE相较于其他竞争方法具有显著优势。我们的代码与数据已开源至https://github.com/RecklessRonan/MuSE。