Multimodal Large Language Model (MLLM) recently has been a new rising research hotspot, which uses powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of MLLM, such as writing stories based on images and OCR-free math reasoning, are rare in traditional methods, suggesting a potential path to artificial general intelligence. In this paper, we aim to trace and summarize the recent progress of MLLM. First of all, we present the formulation of MLLM and delineate its related concepts. Then, we discuss the key techniques and applications, including Multimodal Instruction Tuning (M-IT), Multimodal In-Context Learning (M-ICL), Multimodal Chain of Thought (M-CoT), and LLM-Aided Visual Reasoning (LAVR). Finally, we discuss existing challenges and point out promising research directions. In light of the fact that the era of MLLM has only just begun, we will keep updating this survey and hope it can inspire more research. An associated GitHub link collecting the latest papers is available at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.

翻译：多模态大语言模型（MLLM）近期成为新兴的研究热点，其以强大的大语言模型（LLMs）为“大脑”执行多模态任务。MLLM涌现出的惊人能力，如基于图像创作故事、免OCR数学推理等，在传统方法中极为罕见，这暗示着通向通用人工智能的可能路径。本文旨在追溯并总结MLLM的最新进展。首先，我们提出MLLM的数学形式化定义并界定相关概念；继而，探讨包括多模态指令微调（M-IT）、多模态上下文学习（M-ICL）、多模态思维链（M-CoT）及大语言模型辅助视觉推理（LAVR）在内的关键技术与应用；最后，讨论现有挑战并指出有前景的研究方向。鉴于MLLM时代方才开启，我们将持续更新本综述，期冀激发更多研究。相关最新论文的GitHub链接为：https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models。