The evaluation of Long Video Understanding (LVU) performance poses an important but challenging research problem. Despite previous efforts, the existing video understanding benchmarks are severely constrained by several issues, especially the insufficient lengths of videos, a lack of diversity in video types and evaluation tasks, and the inappropriateness for evaluating LVU performances. To address the above problems, we propose a new benchmark, called MLVU (Multi-task Long Video Understanding Benchmark), for the comprehensive and in-depth evaluation of LVU. MLVU presents the following critical values: 1) The substantial and flexible extension of video lengths, which enables the benchmark to evaluate LVU performance across a wide range of durations. 2) The inclusion of various video genres, e.g., movies, surveillance footage, egocentric videos, cartoons, game videos, etc., which reflects the models' LVU performances in different scenarios. 3) The development of diversified evaluation tasks, which enables a comprehensive examination of MLLMs' key abilities in long-video understanding. The empirical study with 20 latest MLLMs reveals significant room for improvement in today's technique, as all existing methods struggle with most of the evaluation tasks and exhibit severe performance degradation when handling longer videos. Additionally, it suggests that factors such as context length, image-understanding quality, and the choice of LLM backbone can play critical roles in future advancements. We anticipate that MLVU will advance the research of long video understanding by providing a comprehensive and in-depth analysis of MLLMs.

翻译：长视频理解（LVU）性能评估是一个重要但具有挑战性的研究问题。尽管已有诸多努力，现有视频理解基准仍受到若干问题的严重制约，尤其是视频长度不足、视频类型与评估任务缺乏多样性，以及不适用于评估LVU性能。为解决上述问题，我们提出了一项名为MLVU（多任务长视频理解基准）的新基准，用于全面深入地评估LVU。MLVU具有以下关键价值：1）大幅且灵活地扩展视频长度，使基准能够评估不同时长下的LVU性能；2）涵盖多种视频类型，例如电影、监控录像、第一人称视频、动画、游戏视频等，反映模型在不同场景下的LVU表现；3）开发多样化的评估任务，从而全面检验多模态大语言模型（MLLMs）在长视频理解中的关键能力。基于20个最新MLLMs的实证研究表明，当前技术仍有显著提升空间——所有现有方法在大多数评估任务中均表现挣扎，且在处理较长视频时性能严重下降。此外，研究提示上下文长度、图像理解质量以及LLM主干网络的选择等要素可能在未来发展中将发挥关键作用。我们期待MLVU能通过对MLLMs的全面深入分析，推动长视频理解研究的进步。