Many recent studies leverage the pre-trained CLIP for text-video cross-modal retrieval by tuning the backbone with additional heavy modules, which not only brings huge computational burdens with much more parameters, but also leads to the knowledge forgetting from upstream models.In this work, we propose the VoP: Text-Video Co-operative Prompt Tuning for efficient tuning on the text-video retrieval task. The proposed VoP is an end-to-end framework with both video & text prompts introducing, which can be regarded as a powerful baseline with only 0.1% trainable parameters. Further, based on the spatio-temporal characteristics of videos, we develop three novel video prompt mechanisms to improve the performance with different scales of trainable parameters. The basic idea of the VoP enhancement is to model the frame position, frame context, and layer function with specific trainable prompts, respectively. Extensive experiments show that compared to full fine-tuning, the enhanced VoP achieves a 1.4% average R@1 gain across five text-video retrieval benchmarks with 6x less parameter overhead. The code will be available at https://github.com/bighuang624/VoP.

翻译：近期诸多研究通过引入额外重型模块对预训练CLIP模型进行主干网络微调，以实现文本-视频跨模态检索。然而，该方法不仅因参数量激增带来巨大计算负担，还可能导致上游模型知识遗忘。为此，本文提出VoP：面向文本-视频检索任务的文本-视频协同提示微调方法。VoP是一个端到端框架，通过同时引入视频与文本提示，仅需0.1%可训练参数即可构建强基线模型。进一步，基于视频的时空特性，我们开发了三种新型视频提示机制，可在不同规模可训练参数下提升性能。VoP增强的核心思想是分别利用特定可训练提示对帧位置、帧上下文及层级功能进行建模。大量实验表明，相较于全参数微调，增强版VoP在五个文本-视频检索基准上平均R@1提升1.4%，同时参数量降低至1/6。代码将发布于https://github.com/bighuang624/VoP。