Multimodal representation learning poses significant challenges in capturing informative and distinct features from multiple modalities. Existing methods often struggle to exploit the unique characteristics of each modality due to unified multimodal annotations. In this study, we propose Self-MI in the self-supervised learning fashion, which also leverage Contrastive Predictive Coding (CPC) as an auxiliary technique to maximize the Mutual Information (MI) between unimodal input pairs and the multimodal fusion result with unimodal inputs. Moreover, we design a label generation module, $ULG_{MI}$ for short, that enables us to create meaningful and informative labels for each modality in a self-supervised manner. By maximizing the Mutual Information, we encourage better alignment between the multimodal fusion and the individual modalities, facilitating improved multimodal fusion. Extensive experiments on three benchmark datasets including CMU-MOSI, CMU-MOSEI, and SIMS, demonstrate the effectiveness of Self-MI in enhancing the multimodal fusion task.

翻译：多模态表示学习在从多个模态中捕获信息丰富且独特的特征方面面临重大挑战。现有方法通常因统一的多模态标注而难以充分利用每种模态的独特特性。在本研究中，我们提出以自监督学习方式设计的Self-MI方法，该方法还利用对比预测编码（CPC）作为辅助技术，最大化单模态输入对与基于单模态输入的多模态融合结果之间的互信息（MI）。此外，我们设计了一个标签生成模块（简称$ULG_{MI}$），能够以自监督方式为每种模态生成有意义且信息丰富的标签。通过最大化互信息，我们促进了多模态融合结果与各独立模态之间更好的对齐，从而改善多模态融合性能。在CMU-MOSI、CMU-MOSEI和SIMS三个基准数据集上的大量实验表明，Self-MI在增强多模态融合任务方面具有有效性。