Vision-language models (VLMs) are commonly trained by directly inserting image tokens from a pretrained vision encoder into the text stream of a language model. This allows text and image information to fully attend to one another within the model, but becomes rapidly costly for long multi-image conversations or streaming video applications, both in terms of memory and compute. VLMs leveraging cross-attention (CA) are an efficient alternative to token insertion as image tokens are not added to the KV cache. Despite being introduced early on, multimodal CA models are scarce in the current VLM literature and often underperform their token insertion counterparts. In this work, we reinvestigate the effectiveness of cross-attention for vision-language modeling: (i) We analyze the core differences between the cross-attention and self-attention mechanisms, (ii) we train cross-attention VLMs both from a text-only LLM and by adapting a pretrained insertion-based VLM, showing that simple cross-attention is far more competitive with token insertion than previously reported, and (iii) we demonstrate the practical advantages of cross-attention on real-time video captioning, where it naturally maintains low latency and near-constant memory cost. For samples and code, please see our project page at https://kyutai.org/casa .

翻译：视觉-语言模型通常通过将预训练视觉编码器提取的图像标记直接插入语言模型的文本流中进行训练。这种方法允许文本与图像信息在模型内充分交互，但在处理长序列多图像对话或流式视频应用时，会迅速产生高昂的内存与计算成本。利用交叉注意力的视觉-语言模型因其无需将图像标记加入键值缓存而成为标记插入方案的高效替代方案。尽管交叉注意力机制提出较早，当前视觉-语言模型研究中采用多模态交叉注意力的模型仍较为稀缺，且性能常逊于标记插入式模型。本研究重新评估了交叉注意力在视觉-语言建模中的有效性：（i）系统分析了交叉注意力与自注意力机制的核心差异；（ii）分别从纯文本大语言模型出发训练交叉注意力视觉-语言模型，以及通过适配预训练的标记插入式视觉-语言模型进行训练，证明简单的交叉注意力机制与标记插入方法的性能差距远小于既往认知；（iii）在实时视频描述任务中展示了交叉注意力的实际优势——其天然具备低延迟特性并能维持近乎恒定的内存开销。相关示例与代码请访问项目页面：https://kyutai.org/casa。