Adopting contrastive image-text pretrained models like CLIP towards video classification has gained attention due to its cost-effectiveness and competitive performance. However, recent works in this area face a trade-off. Finetuning the pretrained model to achieve strong supervised performance results in low zero-shot generalization. Similarly, freezing the backbone to retain zero-shot capability causes significant drop in supervised accuracy. Because of this, recent works in literature typically train separate models for supervised and zero-shot action recognition. In this work, we propose a multimodal prompt learning scheme that works to balance the supervised and zero-shot performance under a single unified training. Our prompting approach on the vision side caters for three aspects: 1) Global video-level prompts to model the data distribution; 2) Local frame-level prompts to provide per-frame discriminative conditioning; and 3) a summary prompt to extract a condensed video representation. Additionally, we define a prompting scheme on the text side to augment the textual context. Through this prompting scheme, we can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting. By keeping the pretrained backbone frozen, we optimize a much lower number of parameters and retain the existing general representation which helps achieve the strong zero-shot performance. Our codes/models are released at https://github.com/TalalWasim/Vita-CLIP.

翻译：采用对比式图文预训练模型（如CLIP）进行视频分类因其成本效益和竞争力表现而备受关注。然而，该领域的近期研究面临着一个权衡问题：对预训练模型进行微调以提高监督性能会导致零样本泛化能力下降；而冻结主干网络以保持零样本能力则会造成监督准确率显著降低。因此，现有文献通常分别训练用于监督学习和零样本动作识别的独立模型。本研究提出一种多模态提示学习方案，旨在通过统一训练框架平衡监督性能与零样本性能。在视觉侧，我们的提示方法涵盖三个维度：1）全局视频级提示以建模数据分布；2）局部帧级提示以提供逐帧判别性条件；3）摘要提示以提取压缩视频表示。此外，我们在文本侧定义了一种提示方案以增强文本上下文。通过该提示方案，我们在Kinetics-600、HMDB51和UCF101数据集上实现了最优的零样本性能，同时保持监督学习的竞争力。通过冻结预训练骨干网络，我们优化了更少的参数量并保留了现有通用表示，这有助于实现强大的零样本性能。我们的代码/模型已发布于https://github.com/TalalWasim/Vita-CLIP。