Recent contrastive language image pre-training has led to learning highly transferable and robust image representations. However, adapting these models to video domains with minimal supervision remains an open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt an image CLIP model to the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating in an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. We introduce two train objectives, concept distillation and concept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.

翻译：近期对比性语言-图像预训练方法使得学习到的图像表示具有高度可迁移性与鲁棒性。然而，如何以最小监督将这些模型适配至视频领域仍是一个未决问题。本文探索了一条简洁的解决路径：利用语言驱动的自监督学习，将图像CLIP模型适配至视频领域。经时序建模优化后的骨干网络在自蒸馏框架下进行训练，其训练目标在动作概念空间中执行。该空间通过语言编码器从相关文本提示中提取各类动作概念的特征向量构建而成。我们提出两项训练目标——概念蒸馏与概念对齐，既保留原始表示的通用性，又强化了动作及其属性之间的关联。本方法在三个动作识别基准上显著提升了零样本与线性探测性能。