Video Large Language Models (VideoLLMs) have made significant strides in video understanding but struggle with long videos due to the limitations of their backbone LLMs. Existing solutions rely on length extrapolation, which is memory-constrained, or visual token compression, which primarily leverages low-level temporal redundancy while overlooking the more effective high-level knowledge redundancy. To address this, we propose $\textbf{ReTaKe}$, a training-free method with two novel modules DPSelect and PivotKV, to jointly reduce both temporal visual redundancy and knowledge redundancy for video compression. To align with the way of human temporal perception, DPSelect identifies keyframes based on inter-frame distance peaks. To leverage LLMs' learned prior knowledge, PivotKV marks the keyframes as pivots and compress non-pivot frames by pruning low-attention tokens in their KV cache. ReTaKe enables VideoLLMs to process 8 times longer frames (up to 2048), outperforming similar-sized models by 3-5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench. Moreover, by overlapping compression operations with prefilling, ReTaKe introduces only ~10% prefilling latency overhead while reducing decoding latency by ~20%. Our code is available at https://github.com/SCZwangxiao/video-ReTaKe.

翻译：视频大语言模型（VideoLLMs）在视频理解方面取得了显著进展，但由于其骨干大语言模型的限制，在处理长视频时仍面临困难。现有解决方案依赖于长度外推（受内存限制）或视觉令牌压缩（主要利用低层次的时间冗余，而忽视了更有效的高层次知识冗余）。为解决此问题，我们提出了 $\textbf{ReTaKe}$，一种无需训练的方法，包含两个新颖模块 DPSelect 和 PivotKV，以联合减少视频压缩中的时间视觉冗余和知识冗余。为契合人类时间感知的方式，DPSelect 基于帧间距离峰值识别关键帧。为利用大语言模型已习得的先验知识，PivotKV 将关键帧标记为枢轴，并通过修剪非枢轴帧在 KV 缓存中的低注意力令牌来压缩它们。ReTaKe 使 VideoLLMs 能够处理长达 8 倍的帧数（最多 2048 帧），在 VideoMME、MLVU、LongVideoBench 和 LVBench 基准测试中，性能超越同等规模模型 3-5%，甚至可与更大模型相媲美。此外，通过将压缩操作与预填充阶段重叠，ReTaKe 仅引入约 10% 的预填充延迟开销，同时将解码延迟降低约 20%。我们的代码可在 https://github.com/SCZwangxiao/video-ReTaKe 获取。