Pre-trained large language models have recently achieved ground-breaking performance in a wide variety of language understanding tasks. However, the same model can not be applied to multimodal behavior understanding tasks (e.g., video sentiment/humor detection) unless non-verbal features (e.g., acoustic and visual) can be integrated with language. Jointly modeling multiple modalities significantly increases the model complexity, and makes the training process data-hungry. While an enormous amount of text data is available via the web, collecting large-scale multimodal behavioral video datasets is extremely expensive, both in terms of time and money. In this paper, we investigate whether large language models alone can successfully incorporate non-verbal information when they are presented in textual form. We present a way to convert the acoustic and visual information into corresponding textual descriptions and concatenate them with the spoken text. We feed this augmented input to a pre-trained BERT model and fine-tune it on three downstream multimodal tasks: sentiment, humor, and sarcasm detection. Our approach, TextMI, significantly reduces model complexity, adds interpretability to the model's decision, and can be applied for a diverse set of tasks while achieving superior (multimodal sarcasm detection) or near SOTA (multimodal sentiment analysis and multimodal humor detection) performance. We propose TextMI as a general, competitive baseline for multimodal behavioral analysis tasks, particularly in a low-resource setting.

翻译：摘要：预训练大语言模型近期在多种语言理解任务中取得了突破性表现。然而，此类模型无法直接应用于多模态行为理解任务（如视频情感/幽默检测），除非将非语言特征（如声学与视觉特征）与语言信息相结合。多模态联合建模会显著增加模型复杂度，并使训练过程对数据需求激增。尽管网络上有海量文本数据可供使用，但大规模多模态行为视频数据的采集在时间和资金上均极为昂贵。本文探究了当非语言信息以文本形式呈现时，大语言模型能否独立有效地整合这些信息。我们提出将声学与视觉信息转化为对应的文本描述，并与口语文本拼接。将此增强输入馈入预训练的BERT模型，并在三个下游多模态任务（情感检测、幽默检测与讽刺检测）上进行微调。我们的方法TextMI显著降低了模型复杂度，增强了模型决策的可解释性，可应用于多种任务，并在多模态讽刺检测上达到超越现有最优水平的性能，在多模态情感分析和幽默检测上接近最优水平。我们将TextMI作为多模态行为分析任务（尤其在低资源场景下）的通用且具竞争力的基线模型。