This paper proposes Video-Teller, a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment to significantly enhance the video-to-text generation task. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions. To effectively integrate visual and auditory information, Video-Teller builds upon the image-based BLIP-2 model and introduces a cascaded Q-Former which fuses information across frames and ASR texts. To better guide video summarization, we introduce a fine-grained modality alignment objective, where the cascaded Q-Former's output embedding is trained to align with the caption/summary embedding created by a pretrained text auto-encoder. Experimental results demonstrate the efficacy of our proposed video-language foundation model in accurately comprehending videos and generating coherent and precise language descriptions. It is worth noting that the fine-grained alignment enhances the model's capabilities (4% improvement of CIDEr score on MSR-VTT) with only 13% extra parameters in training and zero additional cost in inference.

翻译：本文提出Video-Teller，一种利用多模态融合与细粒度模态对齐显著提升视频到文本生成任务的视频语言基础模型。Video-Teller通过使用冻结的预训练视觉和语言模块提升了训练效率。它借助大型语言模型强大的语言能力，能够生成简洁或详尽的视频描述。为有效整合视觉与听觉信息，Video-Teller基于图像型BLIP-2模型，引入了级联Q-Former结构，该结构跨帧和ASR文本融合信息。为更好地引导视频摘要生成，我们提出细粒度模态对齐目标，通过训练级联Q-Former的输出嵌入与预训练文本自编码器生成的标题/摘要嵌入对齐。实验结果表明，我们提出的视频语言基础模型能够准确理解视频并生成连贯、精确的语言描述。值得注意的是，细粒度对齐仅增加13%的训练参数且零额外推理成本，即可提升模型性能（在MSR-VTT数据集上CIDEr得分提高4%）。