With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey

翻译：随着对通用深度模型的迫切需求，各类预训练大模型相继被提出，如BERT、ViT、GPT等。受这些模型在单一领域（如计算机视觉和自然语言处理）成功的启发，多模态预训练大模型近年来也获得了越来越多的关注。本文对这些模型进行了全面综述，旨在提供新见解并帮助初入领域的研究者追踪最前沿的工作。具体而言，我们首先通过回顾传统深度学习、自然语言处理、计算机视觉和语音领域的预训练工作，介绍多模态预训练的背景。随后，我们阐述多模态预训练模型的任务定义、关键挑战与优势，并聚焦于数据、目标函数、网络架构以及知识增强预训练等方面进行讨论。接下来，我们介绍用于验证大规模多模态预训练模型的下游任务，包括生成、分类和回归任务。同时，我们对典型下游任务的模型参数和结果进行可视化与分析。最后，我们指出该主题未来可能的研究方向，以期为后续工作提供参考。此外，我们持续维护一个关于大规模预训练多模态大模型的论文列表，可通过https://github.com/wangxiao5791509/MultiModal_BigModels_Survey 访问。