Since the resurgence of deep learning, vision-language models (VLMs) enhanced by large language models (LLMs) have grown exponentially in popularity. However, while LLMs can utilize extensive background knowledge and task information with in-context learning, most VLMs still struggle with understanding complex multi-modal prompts with multiple images, making VLMs less effective in downstream vision-language tasks. In this paper, we address the limitation above by 1) introducing vision-language Model with Multi-Modal In-Context Learning(MMICL), a new approach to allow the VLM to deal with multi-modal inputs efficiently; 2) proposing a novel context scheme to augment the in-context learning ability of the VLM; 3) constructing the Multi-modal In-Context Learning (MIC) dataset, designed to enhance the VLM's ability to understand complex multi-modal prompts. Our experiments confirm that MMICL achieves new state-of-the-art zero-shot performance on a wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Our analysis demonstrates that MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges the impressive ICL ability. Furthermore, we observe that MMICL successfully alleviates language bias in VLMs, a common issue for VLMs that often leads to hallucination when faced with extensive textual context. Our code, dataset, dataset tool, and model are available at https://github.com/PKUnlp-icler/MIC

翻译：自深度学习复兴以来，由大语言模型增强的视觉语言模型在流行度上呈指数级增长。然而，尽管大语言模型能够利用广泛的背景知识和任务信息进行上下文学习，大多数视觉语言模型在处理包含多张图像的复杂多模态提示时仍存在困难，导致其在下游视觉语言任务中效果不佳。本文针对上述局限性进行了改进：1）提出了具有多模态上下文学习能力的视觉语言模型（MMICL），这是一种使视觉语言模型高效处理多模态输入的新方法；2）提出了一种新颖的上下文方案以增强视觉语言模型的上下文学习能力；3）构建了多模态上下文学习数据集，旨在提升视觉语言模型理解复杂多模态提示的能力。实验证明，MMICL在多种通用视觉语言任务上取得了新的零样本最佳性能，尤其是在包括MME和MMBbench在内的复杂基准测试中。我们的分析表明，MMICL有效解决了复杂多模态提示理解的挑战，并展现出令人印象深刻的上下文学习能力。此外，我们观察到MMICL成功缓解了视觉语言模型的语言偏差——这一常见问题常导致模型在面对大量文本上下文时产生幻觉。我们的代码、数据集、数据处理工具及模型均已开源，地址为https://github.com/PKUnlp-icler/MIC