Designing an effective representation learning method for multimodal sentiment analysis tasks is a crucial research direction. The challenge lies in learning both shared and private information in a complete modal representation, which is difficult with uniform multimodal labels and a raw feature fusion approach. In this work, we propose a deep modal shared information learning module based on the covariance matrix to capture the shared information between modalities. Additionally, we use a label generation module based on a self-supervised learning strategy to capture the private information of the modalities. Our module is plug-and-play in multimodal tasks, and by changing the parameterization, it can adjust the information exchange relationship between the modes and learn the private or shared information between the specified modes. We also employ a multi-task learning strategy to help the model focus its attention on the modal differentiation training data. We provide a detailed formulation derivation and feasibility proof for the design of the deep modal shared information learning module. We conduct extensive experiments on three common multimodal sentiment analysis baseline datasets, and the experimental results validate the reliability of our model. Furthermore, we explore more combinatorial techniques for the use of the module. Our approach outperforms current state-of-the-art methods on most of the metrics of the three public datasets.

翻译：设计有效的多模态情感分析任务表示学习方法是一个关键研究方向。其难点在于学习完整模态表示中的共享与私有信息，而统一的多模态标签与原始特征融合方法难以实现这一目标。本文提出基于协方差矩阵的深度模态共享信息学习模块，用于捕捉模态间的共享信息；同时采用基于自监督学习策略的标签生成模块，获取模态的私有信息。该模块可即插即用于多模态任务，通过改变参数化方式，能够调整模态间的信息交换关系，并学习指定模态间的私有或共享信息。我们还采用多任务学习策略，帮助模型将注意力聚焦于模态差异化训练数据。针对深度模态共享信息学习模块的设计，我们提供了详细的公式推导与可行性证明。在三个常用多模态情感分析基准数据集上开展广泛实验，实验结果验证了模型可靠性。此外，我们探索了更多模块组合使用技术。在三个公共数据集的大部分指标上，我们的方法均优于当前最先进的方法。