Recent contrastive language image pre-training has led to learning highly transferable and robust image representations. However, adapting these models to video domains with minimal supervision remains an open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt an image CLIP model to the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating in an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. We introduce two train objectives, concept distillation and concept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.

翻译：最近的对比语言-图像预训练方法已学习到高度可迁移且鲁棒的图像表示。然而，如何以极少的监督将这些模型适配到视频领域仍是一个开放性问题。我们探索了该方向的一个简单步骤：利用与语言结合的自监督学习，将图像CLIP模型适配到视频领域。在自蒸馏设置下，一个为时序建模改进的骨干网络通过基于动作概念空间的训练目标进行训练。从语言编码器中利用相关文本提示提取的各种动作概念的特征向量构建了这一空间。我们引入了两个训练目标——概念蒸馏和概念对齐——在保留原始表示通用性的同时，强化了动作及其属性之间的关系。我们的方法在三个动作识别基准上提升了零样本和线性探测性能。