Large language models (LLMs) have enabled the creation of multi-modal LLMs that exhibit strong comprehension of visual data such as images and videos. However, these models usually rely on extensive visual tokens from visual encoders, leading to high computational demands, which limits their applicability in resource-constrained environments and for long-context tasks. In this work, we propose a training-free adaptive inference method for multi-modal LLMs that can accommodate a broad range of efficiency requirements with a minimum performance drop. Our method consists of a) iterative token merging based on embedding similarity before LLMs, and b) progressive token pruning within LLM layers based on multi-modal importance. With a minimalist design, our method can be applied to both video and image LLMs. Extensive experiments on diverse video and image benchmarks demonstrate that, our method substantially reduces computation load (e.g., a $\textbf{7-fold}$ reduction in FLOPs) while preserving the performance of video and image LLMs. Further, under a similar computational cost, our method outperforms the state-of-the-art methods in long video understanding (e.g., $\textbf{+4.6}$ on MLVU). Additionally, our in-depth analysis provides insights into token redundancy and LLM layer behaviors, offering guidance for future research in designing efficient multi-modal LLMs. Our code will be available at https://github.com/LaVi-Lab/AIM.

翻译：大语言模型（LLMs）推动了多模态大语言模型的发展，使其展现出对图像、视频等视觉数据的强大理解能力。然而，这些模型通常依赖视觉编码器产生的大量视觉令牌，导致计算需求高昂，从而限制了其在资源受限环境及长上下文任务中的应用。本研究提出一种免训练的自适应推理方法，可在性能损失最小化的前提下满足广泛的计算效率需求。该方法包含两个核心机制：a) 在输入大语言模型前基于嵌入相似度的迭代令牌合并；b)在大语言模型层内基于多模态重要性的渐进式令牌剪枝。凭借极简的设计，本方法可同时适用于视频与图像大语言模型。在多样化视频与图像基准测试上的大量实验表明，该方法在保持视频与图像大语言模型性能的同时，显著降低了计算负载（例如实现$\textbf{7倍}$的浮点运算量削减）。此外，在相近计算成本下，本方法在长视频理解任务上优于现有最优方法（如在MLVU基准上提升$\textbf{+4.6}$分）。深入的模型分析进一步揭示了令牌冗余现象与大语言模型层的行为特性，为未来设计高效多模态大语言模型提供了理论指导。代码已开源：https://github.com/LaVi-Lab/AIM。