In this paper, we propose a new Multimodal Representation Learning (MRL) method for Multimodal Sentiment Analysis (MSA), which facilitates the adaptive interaction between modalities through Cooperative Sentiment Agents, named Co-SA. Co-SA comprises two critical components: the Sentiment Agents Establishment (SAE) phase and the Sentiment Agents Cooperation (SAC) phase. During the SAE phase, each sentiment agent deals with an unimodal signal and highlights explicit dynamic sentiment variations within the modality via the Modality-Sentiment Disentanglement (MSD) and Deep Phase Space Reconstruction (DPSR) modules. Subsequently, in the SAC phase, Co-SA meticulously designs task-specific interaction mechanisms for sentiment agents so that coordinating multimodal signals to learn the joint representation. Specifically, Co-SA equips an independent policy model for each sentiment agent that captures significant properties within the modality. These policies are optimized mutually through the unified reward adaptive to downstream tasks. Benefitting from the rewarding mechanism, Co-SA transcends the limitation of pre-defined fusion modes and adaptively captures unimodal properties for MRL in the multimodal interaction setting. To demonstrate the effectiveness of Co-SA, we apply it to address Multimodal Sentiment Analysis (MSA) and Multimodal Emotion Recognition (MER) tasks. Our comprehensive experimental results demonstrate that Co-SA excels at discovering diverse cross-modal features, encompassing both common and complementary aspects. The code can be available at https://github.com/smwanghhh/Co-SA.

翻译：本文提出了一种面向多模态情感分析（MSA）的新型多模态表征学习方法（MRL），该方法通过协作式情感智能体（Co-SA）实现模态间的自适应交互。Co-SA包含两个关键组件：情感智能体建立（SAE）阶段与情感智能体协作（SAC）阶段。在SAE阶段，每个情感智能体处理单一模态信号，并借助模态-情感解耦（MSD）与深度相空间重构（DPSR）模块突出模态内的显式动态情感变化。随后在SAC阶段，Co-SA为情感智能体精心设计了任务特定交互机制，从而协调多模态信号以学习联合表征。具体而言，Co-SA为每个情感智能体配备独立策略模型以捕获模态内的关键属性，这些策略通过适应下游任务的统一奖励机制实现相互优化。得益于该奖励机制，Co-SA突破了预定义融合模式的限制，在多模态交互场景中自适应地捕获单模态属性以实现MRL。为验证Co-SA的有效性，我们将其应用于多模态情感分析（MSA）与多模态情绪识别（MER）任务。综合实验结果表明，Co-SA在发现包含共性与互补维度的多样化跨模态特征方面表现优异。相关代码已开源至https://github.com/smwanghhh/Co-SA。