The recent progress in Large Language Models (LLM) has spurred various advancements in image-language conversation agents, while how to build a proficient video-based dialogue system is still under exploration. Considering the extensive scale of LLM and visual backbone, minimal GPU memory is left for facilitating effective temporal modeling, which is crucial for comprehending and providing feedback on videos. To this end, we propose Branching Temporal Adapter (BT-Adapter), a novel method for extending image-language pretrained models into the video domain. Specifically, BT-Adapter serves as a plug-and-use temporal modeling branch alongside the pretrained visual encoder, which is tuned while keeping the backbone frozen. Just pretrained once, BT-Adapter can be seamlessly integrated into all image conversation models using this version of CLIP, enabling video conversations without the need for video instructions. Besides, we develop a unique asymmetric token masking strategy inside the branch with tailor-made training tasks for BT-Adapter, facilitating faster convergence and better results. Thanks to BT-Adapter, we are able to empower existing multimodal dialogue models with strong video understanding capabilities without incurring excessive GPU costs. Without bells and whistles, BT-Adapter achieves (1) state-of-the-art zero-shot results on various video tasks using thousands of fewer GPU hours. (2) better performance than current video chatbots without any video instruction tuning. (3) state-of-the-art results of video chatting using video instruction tuning, outperforming previous SOTAs by a large margin.

翻译：大型语言模型（LLM）的最新进展推动了图像语言对话代理的诸多突破，而如何构建高效的基于视频的对话系统仍处于探索阶段。考虑到LLM与视觉骨干网络的庞大规模，用于促进有效时间建模的GPU内存极为有限，而时间建模恰恰是理解与反馈视频内容的核心。为此，我们提出分支时间适配器（BT-Adapter），一种将图像语言预训练模型扩展至视频领域的新方法。具体而言，BT-Adapter作为即插即用的时间建模分支，附着于预训练视觉编码器旁，在保持骨干网络冻结的同时进行调优。仅需一次预训练，BT-Adapter即可无缝集成至所有采用该版本CLIP的图像对话模型，使其无需视频指令即可实现视频对话。此外，我们在分支内部设计了一种独特的非对称令牌掩码策略，搭配为BT-Adapter定制的训练任务，以加速收敛并提升性能。得益于BT-Adapter，我们能够在不造成过高GPU开销的前提下，为现有多模态对话模型赋予强大的视频理解能力。无需繁杂修饰，BT-Adapter实现了：（1）在多种视频任务上以数千倍更少的GPU小时数取得最先进的零样本结果；（2）无需任何视频指令调优即可超越现有视频聊天机器人的性能；（3）通过视频指令调优取得视频对话的最先进结果，大幅超越此前最佳模型。