Many recent studies leverage the pre-trained CLIP for text-video cross-modal retrieval by tuning the backbone with additional heavy modules, which not only brings huge computational burdens with much more parameters, but also leads to the knowledge forgetting from upstream models. In this work, we propose the VoP: Text-Video Co-operative Prompt Tuning for efficient tuning on the text-video retrieval task. The proposed VoP is an end-to-end framework with both video & text prompts introducing, which can be regarded as a powerful baseline with only 0.1% trainable parameters. Further, based on the spatio-temporal characteristics of videos, we develop three novel video prompt mechanisms to improve the performance with different scales of trainable parameters. The basic idea of the VoP enhancement is to model the frame position, frame context, and layer function with specific trainable prompts, respectively. Extensive experiments show that compared to full fine-tuning, the enhanced VoP achieves a 1.4% average R@1 gain across five text-video retrieval benchmarks with 6x less parameter overhead. The code will be available at https://github.com/bighuang624/VoP.

翻译：许多近期研究通过为预训练的CLIP模型添加额外重型模块进行微调，以完成文本-视频跨模态检索任务，这不仅因参数量剧增带来巨大计算负担，还导致上游模型知识遗忘。本文提出VoP：一种面向文本-视频检索任务的高效微调方法——文本-视频协同提示微调。所提出的VoP是一个端到端框架，通过同时引入视频提示与文本提示，仅需0.1%的可训练参数即可成为强大基线模型。进一步基于视频的时空特性，我们发展出三种新颖的视频提示机制，能够以不同规模的可训练参数提升性能。VoP增强的核心思想是分别建模帧位置、帧上下文及层功能对应的特定可训练提示。大量实验表明，与全参数微调相比，增强版VoP在五个文本-视频检索基准上实现了平均1.4%的R@1指标提升，同时参数开销降低至1/6。代码将开源至https://github.com/bighuang624/VoP。