Conversation agents fueled by Large Language Models (LLMs) are providing a new way to interact with visual data. While there have been initial attempts for image-based conversation models, this work addresses the underexplored field of video-based conversation by introducing Video-ChatGPT. It is a multimodal model that merges a video-adapted visual encoder with a LLM. The model is capable of understanding and generating human-like conversations about videos. We introduce a new dataset of 100,000 video-instruction pairs used to train Video-ChatGPT acquired via manual and semi-automated pipeline that is easily scalable and robust to label noise. We also develop a quantiative evaluation framework for video-based dialogue models to objectively analyse the strengths and weaknesses of proposed models. Our code, models, instruction-sets and demo are released at https://github.com/mbzuai-oryx/Video-ChatGPT.

翻译：以大型语言模型（LLM）为驱动的对话智能体正为人机视觉数据交互开辟新范式。尽管已有初步探索图像对话模型的研究，本文聚焦于未充分开发的多轮视频对话领域，提出Video-ChatGPT模型。该多模态模型通过融合视频适配的视觉编码器与LLM，能够理解并生成类人式的视频内容对话。我们构建了包含10万条视频-指令对的新型数据集来训练Video-ChatGPT，该数据集通过具备良好可扩展性与标注噪声鲁棒性的人工与半自动化流水线构建，同时开发了面向视频对话模型的量化评估框架，可客观分析所提出模型的优劣势。相关代码、模型、指令集与演示系统已在https://github.com/mbzuai-oryx/Video-ChatGPT 开源。