This paper proposes Video-Teller, a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment to significantly enhance the video-to-text generation task. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions. To effectively integrate visual and auditory information, Video-Teller builds upon the image-based BLIP-2 model and introduces a cascaded Q-Former which fuses information across frames and ASR texts. To better guide video summarization, we introduce a fine-grained modality alignment objective, where the cascaded Q-Former's output embedding is trained to align with the caption/summary embedding created by a pretrained text auto-encoder. Experimental results demonstrate the efficacy of our proposed video-language foundation model in accurately comprehending videos and generating coherent and precise language descriptions. It is worth noting that the fine-grained alignment enhances the model's capabilities (4% improvement of CIDEr score on MSR-VTT) with only 13% extra parameters in training and zero additional cost in inference.

翻译：本文提出Video-Teller这一视频-语言基础模型，通过多模态融合与细粒度模态对齐显著提升视频到文本生成任务。该模型利用冻结的预训练视觉与语言模块提高训练效率，并借助大语言模型强大的语言能力生成简洁或详尽的视频描述。为有效整合视觉与听觉信息，Video-Teller在基于图像的BLIP-2模型基础上引入级联Q-Former，实现跨帧信息与ASR文本的融合。为更好地引导视频摘要生成，我们提出细粒度模态对齐目标：训练级联Q-Former的输出嵌入与预训练文本自编码器生成的标题/摘要嵌入对齐。实验结果表明，所提出的视频-语言基础模型能准确理解视频并生成连贯精准的语言描述。值得注意的是，细粒度对齐仅增加13%的训练参数量且推理零额外成本，即可提升模型性能（MSR-VTT数据集上CIDEr分数提升4%）。