Multimodal sentiment analysis is an important area for understanding the user's internal states. Deep learning methods were effective, but the problem of poor interpretability has gradually gained attention. Previous works have attempted to use attention weights or vector distributions to provide interpretability. However, their explanations were not intuitive and can be influenced by different trained models. This study proposed a novel approach to provide interpretability by converting nonverbal modalities into text descriptions and by using large-scale language models for sentiment predictions. This provides an intuitive approach to directly interpret what models depend on with respect to making decisions from input texts, thus significantly improving interpretability. Specifically, we convert descriptions based on two feature patterns for the audio modality and discrete action units for the facial modality. Experimental results on two sentiment analysis tasks demonstrated that the proposed approach maintained, or even improved effectiveness for sentiment analysis compared to baselines using conventional features, with the highest improvement of 2.49% on the F1 score. The results also showed that multimodal descriptions have similar characteristics on fusing modalities as those of conventional fusion methods. The results demonstrated that the proposed approach is interpretable and effective for multimodal sentiment analysis.

翻译：多模态情感分析是理解用户内部状态的重要领域。深度学习方法虽有效，但其可解释性差的问题逐渐受到关注。以往研究尝试利用注意力权重或向量分布提供可解释性，然而这些解释并不直观，且易受不同训练模型的影响。本研究提出一种新方法，通过将非言语模态转化为文本描述，并利用大规模语言模型进行情感预测，从而提供可解释性。该方法提供了一种直观途径，可直接解释模型基于输入文本做出决策所依赖的内容，因此显著提升了可解释性。具体而言，我们针对音频模态的两种特征模式及面部模态的离散动作单元进行描述转换。两个情感分析任务的实验结果表明，与传统特征基线方法相比，所提方法在情感分析效果上保持甚至有所提升，F1分数最高提升2.49%。结果还表明，多模态描述在模态融合特性上与常规融合方法具有相似性。这些结果证明，所提方法对多模态情感分析具有可解释性和有效性。