Modern multimodal large language models (MLLMs) adopt a unified self-attention design that processes visual and textual tokens at every Transformer layer, incurring substantial computational overhead. In this work, we revisit the necessity of such dense visual processing and show that projected visual embeddings are already well-aligned with the language space, while effective vision-language interaction occurs in only a small subset of layers. Based on these insights, we propose ViCA (Vision-only Cross-Attention), a minimal MLLM architecture in which visual tokens bypass all self-attention and feed-forward layers, interacting with text solely through sparse cross-attention at selected layers. Extensive evaluations across three MLLM backbones, nine multimodal benchmarks, and 26 pruning-based baselines show that ViCA preserves 98% of baseline accuracy while reducing visual-side computation to 4%, consistently achieving superior performance-efficiency trade-offs. Moreover, ViCA provides a regular, hardware-friendly inference pipeline that yields over 3.5x speedup in single-batch inference and over 10x speedup in multi-batch inference, reducing visual grounding to near-zero overhead compared with text-only LLMs. It is also orthogonal to token pruning methods and can be seamlessly combined for further efficiency gains. Our code is available at https://github.com/EIT-NLP/ViCA.

翻译：现代多模态大语言模型（MLLMs）普遍采用统一的自注意力设计，即在每个Transformer层中同时处理视觉与文本标记，这带来了巨大的计算开销。本研究重新审视了这种密集视觉处理的必要性，并证明投影后的视觉嵌入已与语言空间充分对齐，而有效的视觉-语言交互仅发生在少数特定层中。基于此发现，我们提出ViCA（纯视觉交叉注意力）——一种极简的MLLM架构。在该架构中，视觉标记完全绕过所有自注意力层与前馈网络层，仅通过选定层中的稀疏交叉注意力与文本进行交互。通过对三种MLLM骨干网络、九个多模态基准测试及26种基于剪枝的基线方法进行广泛评估，结果表明ViCA在保持基线模型98%精度的同时，将视觉侧计算量降至4%，持续实现更优的性能-效率权衡。此外，ViCA提供了规整且硬件友好的推理流程，在单批次推理中实现超过3.5倍的加速，在多批次推理中实现超过10倍的加速，相较于纯文本大语言模型，其视觉基础处理开销已降至近乎为零。该方法与标记剪枝技术正交，可无缝结合以进一步提升效率。代码已开源：https://github.com/EIT-NLP/ViCA。