This paper undertakes an empirical study to revisit the latest advancements in Multimodal Large Language Models (MLLMs): Video Assistant. This study, namely FreeVA, aims to extend existing image-based MLLM to the video domain in a training-free manner. The study provides an essential, yet must-know baseline, and reveals several surprising findings: 1) FreeVA, leveraging only offline image-based MLLM without additional training, excels in zero-shot video question-answering (e.g., MSVD-QA, ActivityNet-QA, and MSRVTT-QA), even surpassing state-of-the-art methods that involve video instruction tuning. 2) While mainstream video-based MLLMs typically initialize with an image-based MLLM (e.g., LLaVA) and then fine-tune using video instruction tuning, the study indicates that utilizing the widely adopted VideoInstruct-100K for video instruction tuning doesn't actually lead to better performance compared to not training at all. 3) The commonly used evaluation metrics in existing works are significantly influenced by changes in the GPT API version over time. If ignored, this could affect the fairness and uniformity of comparisons between different methods and impact the analysis and judgment of researchers in the field. The advancement of MLLMs is currently thriving, drawing numerous researchers into the field. We aim for this work to serve as a plug-and-play, simple yet effective baseline, encouraging the direct evaluation of existing MLLMs in video domain while also standardizing the field of video conversational models to a certain extent. Also, we encourage researchers to reconsider: Have current video MLLM methods truly acquired knowledge beyond image MLLM? Code is available at https://github.com/whwu95/FreeVA

翻译：本文通过实证研究重新审视多模态大语言模型（MLLM）的最新进展——视频助手。本研究提出的方法FreeVA，旨在以无需训练的方式将现有的基于图像的MLLM扩展至视频领域。该研究提供了一个基础且必须了解基准，并揭示了若干令人意外的发现：1）无需额外训练，仅利用离线图像MLLM的FreeVA在零样本视频问答任务（如MSVD-QA、ActivityNet-QA和MSRVTT-QA）中表现出色，甚至超越了经过视频指令调优的最新方法；2）尽管主流基于视频的MLLM通常以图像MLLM（如LLaVA）为初始化模型，再通过视频指令调优进行微调，但研究表明，采用广泛使用的VideoInstruct-100K数据集进行视频指令调优，实际效果并不优于不进行任何训练；3）现有工作中常用的评估指标会随时间受GPT API版本变化的显著影响。若忽视这一点，可能导致不同方法间比较的公平性和统一性受损，进而影响研究人员的分析与判断。当前MLLM发展迅猛，吸引众多研究者投身该领域。我们期望此工作能作为一个即插即用、简洁而有效的基准，鼓励直接评估现有MLLM在视频领域的表现，同时在一定程度上规范视频对话模型的研究方向。此外，我们呼吁研究者反思：当前视频MLLM方法是否真正获得了超越图像MLLM的知识？代码已开源：https://github.com/whwu95/FreeVA