Multimodal large language models (MLLMs) typically extract visual features from the final layers of a pretrained Vision Transformer (ViT). This widespread deep-layer bias, however, is largely driven by empirical convention rather than principled analysis. While prior studies suggest that different ViT layers capture different types of information, with shallower layers focusing on fine visual details and deeper layers aligning more closely with textual semantics, the impact of this variation on MLLM performance remains underexplored. We present the first comprehensive study of visual layer selection for MLLMs, analyzing representation similarity across ViT layers to establish shallow, middle, and deep layer groupings. Through extensive evaluation of MLLMs (1.4B-7B parameters) across 10 benchmarks encompassing 60+ tasks, we find that while deep layers excel in semantic-rich tasks like OCR, shallow and middle layers significantly outperform them on fine-grained visual tasks including counting, positioning, and object localization. Building on these insights, we propose a lightweight feature fusion method that strategically incorporates shallower layers, achieving consistent improvements over both single-layer and specialized fusion baselines. Our work offers the first principled study of visual layer selection in MLLMs, showing that MLLMs can often see better when they look shallower.

翻译：多模态大语言模型（MLLMs）通常从预训练视觉Transformer（ViT）的最终层提取视觉特征。然而，这种普遍存在的深层偏好主要源于经验性惯例而非原理性分析。尽管先前研究表明ViT的不同层捕获不同类型的信息——浅层关注精细视觉细节，深层更贴合文本语义——但这种差异对MLLM性能的影响仍未得到充分探索。我们首次系统研究了MLLM的视觉层选择策略，通过分析ViT各层的表征相似性，确立了浅层、中层和深层特征分组。在涵盖60余项任务的10个基准测试中对MLLM（1.4B-7B参数）进行广泛评估后发现：虽然深层在OCR等语义密集型任务中表现优异，但浅层和中间层在计数、定位和目标检测等细粒度视觉任务上显著优于深层。基于这些发现，我们提出一种轻量级特征融合方法，通过策略性整合浅层特征，在单层基准和专用融合基线模型上均实现了持续改进。本研究首次为MLLM的视觉层选择提供了原理性论证，表明多模态大语言模型通过观察更浅层的视觉特征往往能获得更优性能。