Recent vision-language models have achieved tremendous progress far beyond what we ever expected. However, their computational costs are also dramatically growing with rapid development, especially for the large models. It makes model acceleration exceedingly critical in a scenario of limited resources. Although extensively studied for unimodal models, the acceleration for multimodal models, especially the vision-language Transformers, is relatively under-explored. To pursue more efficient and accessible vision-language Transformers, this paper introduces \textbf{Cross}-\textbf{G}uided \textbf{E}nsemble of \textbf{T}okens (\textbf{\emph{CrossGET}}), a universal acceleration framework for vision-language Transformers. This framework adaptively combines tokens through real-time, cross-modal guidance, thereby achieving substantial acceleration while keeping high performance. \textit{CrossGET} has two key innovations: 1) \textit{Cross-Guided Matching and Ensemble}. \textit{CrossGET} incorporates cross-modal guided token matching and ensemble to exploit cross-modal information effectively, only introducing cross-modal tokens with negligible extra parameters. 2) \textit{Complete-Graph Soft Matching}. In contrast to the existing bipartite soft matching approach, \textit{CrossGET} introduces a complete-graph soft matching policy to achieve more reliable token-matching results while maintaining parallelizability and high efficiency. Extensive experiments are conducted on various vision-language tasks, including image-text retrieval, visual reasoning, image captioning, and visual question answering. Performance on both classic multimodal architectures and emerging multimodal LLMs demonstrate the effectiveness and versatility of the proposed \textit{CrossGET} framework. The code will be at \url{https://github.com/sdc17/CrossGET}.

翻译：近年来，视觉语言模型取得了远超预期的巨大进展。然而，伴随其快速发展，计算成本也急剧增长，尤其对于大型模型而言。这使得模型加速在资源受限场景中变得至关重要。尽管单模态模型的加速已被广泛研究，但多模态模型（尤其是视觉语言Transformer）的加速方法相对较少被探索。为追求更高效、更易用的视觉语言Transformer，本文提出**跨引导式Token集成**（**CrossGET**），一种适用于视觉语言Transformer的通用加速框架。该框架通过实时的跨模态引导自适应地集成Token，从而在保持高性能的同时实现显著加速。CrossGET包含两项关键创新：1）**跨引导匹配与集成**：CrossGET引入跨模态引导的Token匹配与集成，有效利用跨模态信息，仅以可忽略的额外参数引入跨模态Token；2）**完全图软匹配**：与现有二分图软匹配方法不同，CrossGET提出完全图软匹配策略，在保持并行性与高效率的同时，实现更可靠的Token匹配结果。我们在多种视觉语言任务（包括图像-文本检索、视觉推理、图像描述和视觉问答）上进行了广泛实验。在经典多模态架构与新兴多模态大语言模型上的性能均验证了所提CrossGET框架的有效性与通用性。代码将发布于\url{https://github.com/sdc17/CrossGET}。