In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.

翻译：本文提出了一种面向多模态理解与生成的视觉-音频-语言全感知预训练模型（VALOR）。与广泛研究的视觉-语言预训练模型不同，VALOR以端到端方式联合建模视觉、音频和语言之间的关系。该模型包含三个独立的单模态表示编码器，以及一个用于多模态条件文本生成的解码器。我们设计了两种预训练任务来训练VALOR模型，包括多模态分组对齐（MGA）和多模态分组描述生成（MGC）。MGA将视觉、语言和音频投影至同一公共空间，同步建立视觉-语言、音频-语言以及视听-语言的对齐关系；MGC则学习在视觉、音频或其联合条件下生成文本标记的能力。为促进视觉-音频-语言预训练研究，我们构建了名为VALOR-1M的大规模高质量三模态数据集，包含100万个附带人工标注视听描述的可听视频。大量实验表明，VALOR能够学习到强大的多模态关联，并泛化至各类下游任务（如检索、描述生成和问答），同时支持不同输入模态（如视觉-语言、音频-语言和视听-语言）。VALOR在多个公开跨模态基准任务上取得了最新的最佳性能。代码与数据可从项目页面 https://casia-iva-group.github.io/projects/VALOR 获取。