With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey. This paper has been published by the journal Machine Intelligence Research (MIR), https://link.springer.com/article/10.1007/s11633-022-1410-8, DOI: 10.1007/s11633-022-1410-8, vol. 20, no. 4, pp. 447-482, 2023.

翻译：随着对通用深度学习模型的迫切需求，诸多预训练大模型如BERT、ViT、GPT等相继被提出。受这些模型在单领域（如计算机视觉和自然语言处理）中成功的启发，多模态预训练大模型近年来也日益受到关注。本文对这些模型进行了全面综述，旨在提供新见解并帮助新研究者追踪最前沿工作。具体而言，我们首先回顾传统深度学习、自然语言处理、计算机视觉和语音领域的预训练工作，介绍多模态预训练的背景。随后，阐述多模态预训练模型（MM-PTMs）的任务定义、关键挑战及优势，并从数据、目标函数、网络架构和知识增强预训练四个维度对MM-PTMs进行重点讨论。在此基础上，介绍用于验证大规模MM-PTMs的下游任务，包括生成、分类和回归任务，并对模型参数及代表性下游任务的结果进行可视化与分析。最后，指出该主题可能促进未来工作的研究方向。此外，我们维护了一份持续更新的预训练多模态大模型论文列表：https://github.com/wangxiao5791509/MultiModal_BigModels_Survey。本文已发表于《Machine Intelligence Research》期刊（MIR），https://link.springer.com/article/10.1007/s11633-022-1410-8，DOI: 10.1007/s11633-022-1410-8，卷20，第4期，第447-482页，2023年。