This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM) designed specifically for video understanding. The model is capable of processing both temporal visual and textual data, making it adept at understanding the complexities of videos. Building upon the success of MiniGPT-v2, which excelled in translating visual features into the LLM space for single images and achieved impressive results on various image-text benchmarks, this paper extends the model's capabilities to process a sequence of frames, enabling it to comprehend videos. MiniGPT4-video does not only consider visual content but also incorporates textual conversations, allowing the model to effectively answer queries involving both visual and text components. The proposed model outperforms existing state-of-the-art methods, registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF, and TVQA benchmarks respectively. Our models and code have been made publicly available here https://vision-cair.github.io/MiniGPT4-video/

翻译：本文介绍了MiniGPT4-Video，一种专门为视频理解设计的多模态大语言模型（LLM）。该模型能够同时处理时间维度的视觉与文本数据，从而擅长理解视频的复杂性。在MiniGPT-v2的成功基础上——该模型善于将视觉特征映射到LLM空间以处理单张图像，并在多项图像-文本基准测试中取得显著成果——本文扩展了模型的能力以处理连续帧序列，使其能够理解视频内容。MiniGPT4-Video不仅考虑视觉内容，还整合了文本对话，使模型能够有效回答涉及视觉和文本组件的查询。所提出的模型优于现有最先进方法，在MSVD、MSRVTT、TGIF和TVQA基准测试中分别实现了4.22%、1.13%、20.82%和13.1%的性能提升。我们的模型和代码已在https://vision-cair.github.io/MiniGPT4-video/ 公开提供。