Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.

翻译：多模态大语言模型（MLLMs）在通用视频理解方面取得了进展，但在处理长时长、高分辨率视频时面临困难——尽管存在显著的时空冗余，它们的视觉变换器（ViTs）或大语言模型仍会平等处理每个像素。我们提出了AutoGaze，一个轻量级模块，用于在视频被ViT或MLLM处理之前移除冗余的图像块。通过下一令牌预测和强化学习进行训练，AutoGaze能够自回归地选择一组最少的、多尺度的图像块，这些图像块可以在用户指定的误差阈值内重建视频，从而在保留信息的同时消除冗余。实验表明，AutoGaze能将视觉令牌数量减少4倍至100倍，并将ViTs和MLLMs的速度提升高达19倍，使得将MLLMs扩展到处理1K帧4K分辨率视频成为可能，并在视频基准测试（例如，在VideoMME上达到67.0%）中取得了优异的结果。此外，我们提出了HLVid：首个包含5分钟4K分辨率视频的高分辨率、长视频问答基准。在该基准上，使用AutoGaze扩展的MLLM比基线模型提升了10.1%，并比之前最好的MLLM高出4.5%。项目页面：https://autogaze.github.io/。