As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench

翻译：随着多模态大语言模型（MLLMs）在处理交错图像-文本数据方面取得进展，评估其少样本学习能力仍是一个开放的挑战。本文介绍了FewMMBench，这是一个为评估MLLMs在少样本条件下表现而设计的综合性基准测试，重点关注上下文学习（ICL）和思维链（CoT）提示。FewMMBench涵盖从属性识别到时序推理的多样化多模态理解任务，支持跨任务类型、模型系列和提示策略的系统性分析。我们评估了来自六个模型系列的26个开源权重MLLMs，涵盖零样本、少样本以及CoT增强的少样本设置。我们的研究结果表明，经过指令调优的模型展现出强大的零样本性能，但在增加演示示例或引入CoT推理时获益甚微，甚至出现性能倒退。基于检索的演示和增加上下文长度带来的提升也有限。这些结果凸显了FewMMBench作为诊断和推进多模态大语言模型少样本学习能力的严谨测试平台的价值。数据可在以下网址获取：https://huggingface.co/datasets/mustafaa/FewMMBench