Pre-trained large language models have recently achieved ground-breaking performance in a wide variety of language understanding tasks. However, the same model can not be applied to multimodal behavior understanding tasks (e.g., video sentiment/humor detection) unless non-verbal features (e.g., acoustic and visual) can be integrated with language. Jointly modeling multiple modalities significantly increases the model complexity, and makes the training process data-hungry. While an enormous amount of text data is available via the web, collecting large-scale multimodal behavioral video datasets is extremely expensive, both in terms of time and money. In this paper, we investigate whether large language models alone can successfully incorporate non-verbal information when they are presented in textual form. We present a way to convert the acoustic and visual information into corresponding textual descriptions and concatenate them with the spoken text. We feed this augmented input to a pre-trained BERT model and fine-tune it on three downstream multimodal tasks: sentiment, humor, and sarcasm detection. Our approach, TextMI, significantly reduces model complexity, adds interpretability to the model's decision, and can be applied for a diverse set of tasks while achieving superior (multimodal sarcasm detection) or near SOTA (multimodal sentiment analysis and multimodal humor detection) performance. We propose TextMI as a general, competitive baseline for multimodal behavioral analysis tasks, particularly in a low-resource setting.

翻译：预训练大语言模型近期在多种语言理解任务中取得了突破性性能。然而，同一模型无法直接应用于多模态行为理解任务（如视频情感/幽默检测），除非非语言特征（如声学特征和视觉特征）能与语言信息融合。联合建模多种模态会显著增加模型复杂度，并使训练过程对数据需求极高。尽管网络上有海量文本数据可用，但收集大规模多模态行为视频数据集在时间和成本上均极为昂贵。本文探究了纯语言模型能否在非语言信息以文本形式呈现时成功整合此类信息。我们提出将声学与视觉信息转化为相应文本描述，并与语音转录文本拼接的方法。将此增强输入馈入预训练BERT模型，并在三个下游多模态任务（情感检测、幽默检测、讽刺检测）中微调。我们提出的TextMI方法显著降低了模型复杂度，增强了模型决策的可解释性，可应用于多样化任务，同时实现了卓越（多模态讽刺检测）或接近现有最优水平（多模态情感分析与多模态幽默检测）的性能。我们将TextMI定位为多模态行为分析任务的通用竞争性基线方法，尤其适用于低资源场景。