This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.

翻译：本文提出MaVEn，一种创新的多粒度视觉编码框架，旨在增强多模态大语言模型在多图像推理任务中的能力。当前的多模态大语言模型主要关注单图像视觉理解，限制了其跨多图像解释与整合信息的能力。MaVEn通过结合离散视觉符号序列（用于抽象粗粒度语义概念）与传统连续表示序列（用于建模细粒度特征），有效解决了这一局限。这种双重方法弥合了视觉数据与文本数据之间的语义鸿沟，从而提升了模型有效处理与解释多图像信息的能力。此外，我们设计了一种针对长序列连续特征的动态约简机制，以提升多图像处理效率。实验结果表明，MaVEn显著增强了多模态大语言模型在复杂多图像场景下的理解能力，同时亦提升了其在单图像场景中的性能表现。