With the rapid development of Large Language Models (LLMs), Video Multi-Modal Large Language Models (Video MLLMs) have achieved remarkable performance in video-language tasks such as video understanding and question answering. However, Video MLLMs face high computational costs, particularly in processing numerous video frames as input, which leads to significant attention computation overhead. A straightforward approach to reduce computational costs is to decrease the number of input video frames. However, simply selecting key frames at a fixed frame rate (FPS) often overlooks valuable information in non-key frames, resulting in notable performance degradation. To address this, we propose Text-guided Video Frame Reduction (TiFRe), a framework that reduces input frames while preserving essential video information. TiFRe uses a Text-guided Frame Sampling (TFS) strategy to select key frames based on user input, which is processed by an LLM to generate a CLIP-style prompt. Pre-trained CLIP encoders calculate the semantic similarity between the prompt and each frame, selecting the most relevant frames as key frames. To preserve video semantics, TiFRe employs a Frame Matching and Merging (FMM) mechanism, which integrates non-key frame information into the selected key frames, minimizing information loss. Experiments show that TiFRe effectively reduces computational costs while improving performance on video-language tasks.

翻译：随着大语言模型（LLMs）的快速发展，视频多模态大语言模型（Video MLLMs）在视频理解、问答等视频-语言任务中取得了显著性能。然而，Video MLLMs面临着高昂的计算成本，尤其是在处理大量视频帧作为输入时，这会导致巨大的注意力计算开销。降低计算成本的一种直接方法是减少输入视频帧的数量。然而，简单地以固定帧率（FPS）选取关键帧往往会忽略非关键帧中的有价值信息，导致明显的性能下降。为解决这一问题，我们提出了文本引导视频帧缩减（TiFRe），这是一个在减少输入帧数的同时保留视频关键信息的框架。TiFRe采用文本引导帧采样（TFS）策略，基于用户输入选择关键帧：用户输入由LLM处理以生成CLIP风格的提示词，预训练的CLIP编码器计算该提示词与每一帧之间的语义相似度，并选择最相关的帧作为关键帧。为保留视频语义，TiFRe采用了帧匹配与融合（FMM）机制，将非关键帧信息整合到选定的关键帧中，从而最小化信息损失。实验表明，TiFRe在有效降低计算成本的同时，提升了视频-语言任务的性能。