Most current audio-visual emotion recognition models lack the flexibility needed for deployment in practical applications. We envision a multimodal system that works even when only one modality is available and can be implemented interchangeably for either predicting emotional attributes or recognizing categorical emotions. Achieving such flexibility in a multimodal emotion recognition system is difficult due to the inherent challenges in accurately interpreting and integrating varied data sources. It is also a challenge to robustly handle missing or partial information while allowing direct switch between regression or classification tasks. This study proposes a versatile audio-visual learning (VAVL) framework for handling unimodal and multimodal systems for emotion regression or emotion classification tasks. We implement an audio-visual framework that can be trained even when audio and visual paired data is not available for part of the training set (i.e., audio only or only video is present). We achieve this effective representation learning with audio-visual shared layers, residual connections over shared layers, and a unimodal reconstruction task. Our experimental results reveal that our architecture significantly outperforms strong baselines on the CREMA-D, MSP-IMPROV, and CMU-MOSEI corpora. Notably, VAVL attains a new state-of-the-art performance in the emotional attribute prediction task on the MSP-IMPROV corpus.

翻译：当前大多数视听情感识别模型缺乏实际应用部署所需的灵活性。我们设想一种多模态系统，即使在仅有一种模态可用时仍能工作，并且能够灵活应用于情感属性预测或分类情感识别任务。由于准确解释和整合多样化数据源存在固有挑战，实现多模态情感识别系统的这种灵活性十分困难。同时，在允许回归与分类任务直接切换的同时，鲁棒地处理缺失或部分信息也是一项挑战。本研究提出了一种通用视听学习（VAVL）框架，用于处理单模态与多模态系统下的情感回归或情感分类任务。我们实现的视听框架即使在部分训练集缺少配对视听数据（即仅存在音频或仅存在视频）的情况下仍可进行训练。我们通过视听共享层、共享层残差连接以及单模态重建任务实现了这种有效的表征学习。实验结果表明，我们的架构在CREMA-D、MSP-IMPROV和CMU-MOSEI数据集上显著优于强基线模型。值得注意的是，VAVL在MSP-IMPROV语料库的情感属性预测任务中取得了新的最优性能。