Pre-training on large-scale video data has become a common recipe for learning transferable spatiotemporal representations in recent years. Despite some progress, existing methods are mostly limited to highly curated datasets (e.g., K400) and exhibit unsatisfactory out-of-the-box representations. We argue that it is due to the fact that they only capture pixel-level knowledge rather than spatiotemporal semantics, which hinders further progress in video understanding. Inspired by the great success of image-text pre-training (e.g., CLIP), we take the first step to exploit language semantics to boost transferable spatiotemporal representation learning. We introduce a new pretext task, Turning to Video for Transcript Sorting (TVTS), which sorts shuffled ASR scripts by attending to learned video representations. We do not rely on descriptive captions and learn purely from video, i.e., leveraging the natural transcribed speech knowledge to provide noisy but useful semantics over time. Our method enforces the vision model to contextualize what is happening over time so that it can re-organize the narrative transcripts, and can seamlessly apply to large-scale uncurated video data in the real world. Our method demonstrates strong out-of-the-box spatiotemporal representations on diverse benchmarks, e.g., +13.6% gains over VideoMAE on SSV2 via linear probing. The code is available at https://github.com/TencentARC/TVTS.

翻译：近年来，在大规模视频数据上进行预训练已成为学习可迁移时空表征的常见方法。尽管取得了一定进展，现有方法大多局限于高度精选的数据集（如K400），且其开箱即用的表征表现不尽如人意。我们认为，这是因为这些方法仅捕获像素级知识而非时空语义，从而阻碍了视频理解的进一步突破。受图像-文本预训练（如CLIP）巨大成功的启发，我们首次尝试利用语言语义来增强可迁移时空表征学习。我们提出了一种新的预文本任务——转向视频进行转录排序（Turning to Video for Transcript Sorting, TVTS），该任务通过关注所学视频表征来对打乱的ASR脚本进行排序。我们不依赖描述性字幕，而是纯粹从视频中学习，即利用自然转录的语音知识随时间提供带有噪声但有用的语义。我们的方法强制视觉模型随时间对事件进行情境化，从而使其能够重新组织叙事转录，并可无缝应用于现实世界中未经策划的大规模视频数据。我们的方法在多个不同基准测试中展现出强大的开箱即用时空表征，例如，在SSV2上通过线性探测相比VideoMAE实现了+13.6%的性能提升。代码已开源：https://github.com/TencentARC/TVTS。