Long Video Question-Answering (LVQA) presents a significant challenge for Multi-modal Large Language Models (MLLMs) due to immense context and overloaded information, which could also lead to prohibitive memory consumption. While existing methods attempt to address these issues by reducing visual tokens or extending model's context length, they may miss useful information or take considerable computation. In fact, when answering given questions, only a small amount of crucial information is required. Therefore, we propose an efficient question-aware memory mechanism, enabling MLLMs to recurrently seek these critical clues. Our approach, named VideoDetective, simplifies this task by iteratively processing video sub-segments. For each sub-segment, a question-aware compression strategy is employed by introducing a few special memory tokens to achieve purposefully compression. This allows models to effectively seek critical clues while reducing visual tokens. Then, due to history context could have a significant impact, we recurrently aggregate and store these memory tokens to update history context, which would be reused for subsequent sub-segments. Furthermore, to more effectively measure model's long video understanding ability, we introduce GLVC (Grounding Long Video Clues), a long video question-answering dataset, which features grounding critical and concrete clues scattered throughout entire videos. Experimental results demonstrate our method enables MLLMs with limited context length of 32K to efficiently process 100K tokens (3600 frames, an hour-long video sampled at 1fps), requiring only 2 minutes and 37GB GPU memory usage. Evaluation results across multiple long video benchmarks illustrate our method can more effectively seek critical clues from massive information.

翻译：长视频问答（LVQA）对多模态大语言模型（MLLMs）提出了重大挑战，原因在于其庞大的上下文和过载的信息，这也可能导致极高的内存消耗。现有方法试图通过减少视觉标记或扩展模型上下文长度来解决这些问题，但可能会遗漏有用信息或需要大量计算。实际上，在回答给定问题时，仅需少量关键信息。因此，我们提出了一种高效的问题感知记忆机制，使MLLMs能够循环搜寻这些关键线索。我们的方法名为VideoDetective，通过迭代处理视频子片段来简化此任务。对于每个子片段，通过引入少量特殊记忆标记，采用问题感知压缩策略以实现有目的的压缩。这使得模型在减少视觉标记的同时，能有效搜寻关键线索。随后，由于历史上下文可能产生显著影响，我们循环聚合并存储这些记忆标记以更新历史上下文，供后续子片段重复使用。此外，为更有效地衡量模型的长视频理解能力，我们引入了GLVC（基于长视频线索的定位）数据集，这是一个长视频问答数据集，其特点在于定位散布在整个视频中的关键且具体的线索。实验结果表明，我们的方法使上下文长度仅为32K的MLLMs能够高效处理100K标记（3600帧，以1fps采样的一小时长视频），仅需2分钟和37GB GPU内存。在多个长视频基准测试中的评估结果表明，我们的方法能更有效地从海量信息中搜寻关键线索。