With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey

翻译：随着对通用深度模型的迫切需求，众多预训练大模型（如BERT、ViT、GPT等）被相继提出。受这些模型在单一领域（如计算机视觉和自然语言处理）成功应用的启发，多模态预训练大模型近年来日益受到关注。本文对这些模型进行了系统综述，旨在提供新的见解，并帮助新兴研究者追踪最前沿的研究工作。具体而言，我们首先通过回顾传统深度学习、自然语言处理、计算机视觉及语音领域的预训练工作，介绍多模态预训练的背景。随后，阐述多模态预训练模型（MM-PTMs）的任务定义、关键挑战与优势，并围绕数据、目标函数、网络架构及知识增强预训练等核心要素展开讨论。接着，介绍用于验证大规模多模态预训练模型的下游任务，包括生成、分类和回归任务，并对代表性下游任务的模型参数与结果进行可视化分析。最后，我们指出该主题可能推动未来研究的潜在方向。此外，我们维护了一份持续更新的大规模多模态预训练模型论文列表：https://github.com/wangxiao5791509/MultiModal_BigModels_Survey