Large language models have demonstrated impressive universal capabilities across a wide range of open-ended tasks and have extended their utility to encompass multimodal conversations. However, existing methods encounter challenges in effectively handling both image and video understanding, particularly with limited visual tokens. In this work, we introduce Chat-UniVi, a unified vision-language model capable of comprehending and engaging in conversations involving images and videos through a unified visual representation. Specifically, we employ a set of dynamic visual tokens to uniformly represent images and videos. This representation framework empowers the model to efficiently utilize a limited number of visual tokens to simultaneously capture the spatial details necessary for images and the comprehensive temporal relationship required for videos. Moreover, we leverage a multi-scale representation, enabling the model to perceive both high-level semantic concepts and low-level visual details. Notably, Chat-UniVi is trained on a mixed dataset containing both images and videos, allowing direct application to tasks involving both mediums without requiring any modifications. Extensive experimental results demonstrate that Chat-UniVi, as a unified model, consistently outperforms even existing methods exclusively designed for either images or videos.

翻译：大语言模型在各类开放式任务中展现出令人瞩目的通用能力，其应用范围已扩展至多模态对话领域。然而现有方法在有效处理图像与视频理解时仍面临挑战，尤其在视觉标记数量有限的情况下。本文提出Chat-UniVi——一种通过统一视觉表示实现图像与视频理解与对话的统一视觉语言模型。具体而言，我们采用动态视觉标记集合对图像与视频进行统一表征。该表征框架使模型能够高效利用有限数量的视觉标记，同时捕获图像所需的空间细节和视频所需的完整时序关系。此外，我们引入多尺度表征机制，使模型既能感知高层语义概念，又能捕捉低层视觉细节。值得注意的是，Chat-UniVi在包含图像与视频的混合数据集上完成训练，可直接应用于涉及两种媒介的任务而无需任何修改。大量实验结果表明，作为统一模型，Chat-UniVi在性能上持续超越现有专门针对图像或视频设计的独立方法。