From image to video understanding, the capabilities of Multi-modal LLMs (MLLMs) are increasingly powerful. However, most existing video understanding benchmarks are relatively short, which makes them inadequate for effectively evaluating the long-sequence modeling capabilities of MLLMs. This highlights the urgent need for a comprehensive and integrated long video understanding benchmark to assess the ability of MLLMs thoroughly. To this end, we propose ALLVB (ALL-in-One Long Video Understanding Benchmark). ALLVB's main contributions include: 1) It integrates 9 major video understanding tasks. These tasks are converted into video QA formats, allowing a single benchmark to evaluate 9 different video understanding capabilities of MLLMs, highlighting the versatility, comprehensiveness, and challenging nature of ALLVB. 2) A fully automated annotation pipeline using GPT-4o is designed, requiring only human quality control, which facilitates the maintenance and expansion of the benchmark. 3) It contains 1,376 videos across 16 categories, averaging nearly 2 hours each, with a total of 252k QAs. To the best of our knowledge, it is the largest long video understanding benchmark in terms of the number of videos, average duration, and number of QAs. We have tested various mainstream MLLMs on ALLVB, and the results indicate that even the most advanced commercial models have significant room for improvement. This reflects the benchmark's challenging nature and demonstrates the substantial potential for development in long video understanding.

翻译：从图像理解到视频理解，多模态大语言模型（MLLMs）的能力日益强大。然而，现有的大多数视频理解基准数据集时长相对较短，这使得它们不足以有效评估MLLMs的长序列建模能力。这凸显了对一个全面、集成的长视频理解基准的迫切需求，以彻底评估MLLMs的能力。为此，我们提出了ALLVB（一体化长视频理解基准）。ALLVB的主要贡献包括：1）它整合了9个主要的视频理解任务。这些任务被转换为视频问答格式，使得单一基准能够评估MLLMs在9种不同视频理解方面的能力，突显了ALLVB的多功能性、全面性和挑战性。2）设计了一个使用GPT-4o的全自动标注流程，仅需人工进行质量控制，这便于基准的维护和扩展。3）它包含16个类别的1,376个视频，平均时长近2小时，总计25.2万个问答对。据我们所知，就视频数量、平均时长和问答对数量而言，它是目前最大的长视频理解基准。我们在ALLVB上测试了各种主流MLLMs，结果表明，即使是最先进的商业模型仍有显著的提升空间。这反映了该基准的挑战性，并展示了长视频理解领域巨大的发展潜力。