Recently, when dealing with high-resolution images, dominant LMMs usually divide them into multiple local images and one global image, which will lead to a large number of visual tokens. In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. This approach not only reduces the number of visual tokens and speeds up inference, but also improves the overall model performance. Specifically, we introduce the following modules based on LLaVA-NeXT: (a) a visual granularity scaler that includes multiple pooling layers to obtain visual tokens with different granularities; (b) a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction. Furthermore, we propose RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data. Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3% reduction in visual tokens and a 2.53$\times$ increase in inference speed on the AI2D benchmark).

翻译：近期，在处理高分辨率图像时，主流的大型多模态模型通常将其划分为多个局部图像和一个全局图像，这会导致视觉令牌数量庞大。本文提出AVG-LLaVA，一种能够根据输入图像和指令自适应选择合适视觉粒度的大型多模态模型。该方法不仅减少了视觉令牌数量并加速推理，还提升了模型的整体性能。具体而言，我们在LLaVA-NeXT基础上引入了以下模块：(a) 视觉粒度缩放器，包含多个池化层以获取不同粒度的视觉令牌；(b) 视觉粒度路由器，包含一个Transformer层、一个MLP层和一个投票层，用于根据图像和指令选择合适的视觉粒度。此外，我们提出了RGLF——一种新颖的训练范式，旨在使路由器预测的粒度与大型多模态模型的偏好对齐，且无需额外的人工标注数据。大量实验与分析表明，AVG-LLaVA在11个基准测试中均取得了优越性能，同时显著减少了视觉令牌数量并加速了推理（例如在AI2D基准测试中视觉令牌减少85.3%，推理速度提升2.53$\times$）。