This work explores an efficient approach to establish a foundational video-text model for tasks including open-vocabulary video classification, text-to-video retrieval, video captioning and video question-answering. We present VideoCoCa that reuses a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with minimal extra training. While previous works adapt image-text models with various cross-frame fusion modules (for example, cross-frame attention layer or perceiver resampler) and finetune the modified architecture on video-text data, we surprisingly find that the generative attentional pooling and contrastive attentional pooling layers in the image-text CoCa design are instantly adaptable to "flattened frame embeddings", yielding a strong zero-shot transfer baseline for many video-text tasks. Specifically, the frozen image encoder of a pretrained image-text CoCa takes each video frame as inputs and generates $N$ token embeddings per frame for totally $T$ video frames. We flatten $(N \times T)$ token embeddings as a long sequence of frozen video representation and apply CoCa's generative attentional pooling and contrastive attentional pooling on top. All model weights including pooling layers are directly loaded from an image-text CoCa pretrained model. Without any video or video-text data, VideoCoCa's zero-shot transfer baseline already achieves state-of-the-art results on zero-shot video classification on Kinetics 400/600/700, UCF101, HMDB51, and Charades, as well as zero-shot text-to-video retrieval on MSR-VTT, ActivityNet Captions and VATEX. We also explore lightweight finetuning on top of VideoCoCa, and achieve strong results on video question-answering (iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA) and video captioning (MSR-VTT, ActivityNet, VATEX, Youcook2). Our approach establishes a simple video-text baseline for future research.

翻译：本文探索了一种高效的基础视频-文本模型构建方法，适用于开放词汇视频分类、文本到视频检索、视频描述生成及视频问答等任务。我们提出VideoCoCa，该模型重复使用预训练的图文对比描述器（CoCa）模型，并通过极少的额外训练将其适配到视频-文本任务中。尽管先前研究通过引入各种跨帧融合模块（如跨帧注意力层或感知器重采样器）来适配图文模型，并在视频-文本数据上微调修改后的架构，我们意外发现图像-文本CoCa设计中的生成性注意力池化与对比性注意力池化层可直接适配于“展平帧嵌入”，从而为多项视频-文本任务提供了强大的零样本迁移基线。具体而言，预训练图像-文本CoCa的冻结图像编码器将每个视频帧作为输入，为总共T个视频帧的每帧生成N个令牌嵌入。我们将(N×T)个令牌嵌入展平为长序列的冻结视频表示，并在此之上应用CoCa的生成性注意力池化与对比性注意力池化。包括池化层在内的所有模型权重均直接加载自预训练的图文CoCa模型。无需任何视频或视频-文本数据，VideoCoCa的零样本迁移基线已在Kinetics 400/600/700、UCF101、HMDB51和Charades的零样本视频分类，以及MSR-VTT、ActivityNet Captions和VATEX的零样本文本到视频检索任务上达到最优结果。我们同时探索了基于VideoCoCa的轻量级微调，在视频问答（iVQA、MSRVTT-QA、MSVD-QA、ActivityNet-QA）和视频描述生成（MSR-VTT、ActivityNet、VATEX、Youcook2）任务上取得了优异表现。所提方法为未来研究建立了简洁的视频-文本基线。