Modern Vision-Language Models (VLMs) achieve impressive performance but are limited by the quadratic complexity of self-attention, which prevents their deployment on edge devices and makes their understanding of high-resolution images and long-context videos prohibitively expensive. To address this challenge, we introduce LinMU (Linear-complexity Multimodal Understanding), a VLM design that achieves linear complexity without using any quadratic-complexity modules while maintaining the performance of global-attention-based VLMs. LinMU replaces every self-attention layer in the VLM with the M-MATE block: a dual-branch module that combines a bidirectional state-space model for global context (Flex-MA branch) with localized Swin-style window attention (Local-Swin branch) for adjacent correlations. To transform a pre-trained VLM into the LinMU architecture, we propose a three-stage distillation framework that (i) initializes both branches with self-attention weights and trains the Flex-MA branch alone, (ii) unfreezes the Local-Swin branch and fine-tunes it jointly with the Flex-MA branch, and (iii) unfreezes the remaining blocks and fine-tunes them using LoRA adapters, while regressing on hidden states and token-level logits of the frozen VLM teacher. On MMMU, TextVQA, LongVideoBench, Video-MME, and other benchmarks, LinMU matches the performance of teacher models, yet reduces Time-To-First-Token (TTFT) by up to 2.7$\times$ and improves token throughput by up to 9.0$\times$ on minute-length videos. Ablations confirm the importance of each distillation stage and the necessity of the two branches of the M-MATE block. The proposed framework demonstrates that state-of-the-art multimodal reasoning can be achieved without quadratic attention, thus opening up avenues for long-context VLMs that can deal with high-resolution images and long videos.

翻译：现代视觉-语言模型（VLMs）虽性能卓越，但受限于自注意力机制的二次复杂度，这阻碍了其在边缘设备上的部署，并使其对高分辨率图像和长上下文视频的理解成本过高。为应对这一挑战，我们提出了LinMU（线性复杂度多模态理解），这是一种VLM设计，在不使用任何二次复杂度模块的同时，实现了线性复杂度，并保持了基于全局注意力的VLMs的性能。LinMU将VLM中的每个自注意力层替换为M-MATE模块：一个双分支结构，结合了用于全局上下文建模的双向状态空间模型（Flex-MA分支）和用于相邻相关性建模的局部Swin风格窗口注意力（Local-Swin分支）。为了将预训练的VLM转换为LinMU架构，我们提出了一个三阶段蒸馏框架：（i）用自注意力权重初始化两个分支并单独训练Flex-MA分支；（ii）解冻Local-Swin分支并与Flex-MA分支联合微调；（iii）解冻其余模块并使用LoRA适配器进行微调，同时对冻结的VLM教师模型的隐藏状态和词元级逻辑值进行回归。在MMMU、TextVQA、LongVideoBench、Video-MME等基准测试中，LinMU与教师模型的性能相当，但在分钟级长度的视频上，首次词元生成时间（TTFT）最多降低了2.7倍，词元吞吐量最多提升了9.0倍。消融实验证实了每个蒸馏阶段的重要性以及M-MATE模块两个分支的必要性。所提出的框架表明，无需二次注意力即可实现最先进的多模态推理，从而为能够处理高分辨率图像和长视频的长上下文VLMs开辟了新途径。