There has been a long-standing quest for a unified audio-visual-text model to enable various multimodal understanding tasks, which mimics the listening, seeing and reading process of human beings. Humans tends to represent knowledge using two separate systems: one for representing verbal (textual) information and one for representing non-verbal (visual and auditory) information. These two systems can operate independently but can also interact with each other. Motivated by this understanding of human cognition, in this paper, we introduce CoAVT -- a novel cognition-inspired Correlated Audio-Visual-Text pre-training model to connect the three modalities. It contains a joint audio-visual encoder that learns to encode audio-visual synchronization information together with the audio and visual content for non-verbal information, and a text encoder to handle textual input for verbal information. To bridge the gap between modalities, CoAVT employs a query encoder, which contains a set of learnable query embeddings, and extracts the most informative audiovisual features of the corresponding text. Additionally, to leverage the correspondences between audio and vision with language respectively, we also establish the audio-text and visual-text bi-modal alignments upon the foundational audiovisual-text tri-modal alignment to enhance the multimodal representation learning. Finally, we jointly optimize CoAVT model with three multimodal objectives: contrastive loss, matching loss and language modeling loss. Extensive experiments show that CoAVT can learn strong multimodal correlations and be generalized to various downstream tasks. CoAVT establishes new state-of-the-art performance on text-video retrieval task on AudioCaps for both zero-shot and fine-tuning settings, audio-visual event classification and audio-visual retrieval tasks on AudioSet and VGGSound.

翻译：长期以来，人们一直致力于构建统一的音视文模型，以支持多种多模态理解任务，这一过程模拟了人类的听觉、视觉与阅读认知机制。人类倾向于使用两个独立系统表征知识：一个负责处理言语（文本）信息，另一个负责处理非言语（视觉与听觉）信息。这两个系统既可独立运作，又能相互交互。受这一人类认知机制的启发，本文提出CoAVT——一种新颖的认知启发的关联音视文预训练模型，旨在连接三种模态。该模型包含一个联合音视频编码器，用于学习编码音视频同步信息以及非言语模态中的音频与视觉内容；同时包含一个文本编码器，用于处理言语模态的文本输入。为弥合模态间的鸿沟，CoAVT采用查询编码器，该编码器包含一组可学习的查询嵌入，用于提取与对应文本最相关的音视频特征。此外，为分别利用音频与视觉同语言之间的对应关系，我们在基础音视文三模态对齐之上，进一步建立音频-文本与视觉-文本双模态对齐，以增强多模态表征学习。最终，我们通过三类多模态目标（对比损失、匹配损失与语言建模损失）联合优化CoAVT模型。大量实验表明，CoAVT能够学习到强大的多模态关联，并可泛化至多种下游任务。在AudioCaps数据集上的文本-视频检索任务中，CoAVT在零样本与微调设置下均取得了当前最优性能；在AudioSet与VGGSound数据集上，其在音视频事件分类与音视频检索任务中亦达到领先水平。