Vision-Language models have shown strong performance in the image-domain -- even in zero-shot settings, thanks to the availability of large amount of pretraining data (i.e., paired image-text examples). However for videos, such paired data is not as abundant. Thus, video-text models are usually designed by adapting pretrained image-text models to video-domain, instead of training from scratch. All such recipes rely on augmenting visual embeddings with temporal information (i.e., image -> video), often keeping text embeddings unchanged or even being discarded. In this paper, we argue that such adapted video-text models can benefit more by augmenting text rather than visual information. We propose VicTR, which jointly-optimizes text and video tokens, generating 'Video-conditioned Text' embeddings. Our method can further make use of freely-available semantic information, in the form of visually-grounded auxiliary text (e.g., object or scene information). We conduct experiments on multiple benchmarks including supervised (Kinetics-400, Charades), zero-shot and few-shot (HMDB-51, UCF-101) settings, showing competitive performance on activity recognition based on video-text models.

翻译：视觉-语言模型在图像领域表现出色——即使在零样本设置下，这得益于大量预训练数据（即配对图像-文本样本）的可用性。然而对于视频而言，此类配对数据并不充足。因此，视频-文本模型通常通过调整预训练的图像-文本模型适应视频领域来设计，而非从头训练。所有这些方法都依赖于用时间信息增强视觉嵌入（即图像→视频），通常保持文本嵌入不变甚至被丢弃。本文中，我们认为此类适应的视频-文本模型通过增强文本而非视觉信息能够获得更多益处。我们提出VicTR，该方法联合优化文本和视频令牌，生成"视频条件文本"嵌入。我们的方法还能进一步利用自由可用的语义信息，其形式为基于视觉的辅助文本（例如对象或场景信息）。我们在多个基准上开展实验，包括监督式（Kinetics-400、Charades）、零样本和少样本（HMDB-51、UCF-101）设置，在基于视频-文本模型的活动识别中展现出竞争力。