Large-scale visual-language pre-trained models have achieved significant success in various video tasks. However, most existing methods follow an "adapt then align" paradigm, which adapts pre-trained image encoders to model video-level representations and utilizes one-hot or text embedding of the action labels for supervision. This paradigm overlooks the challenge of mapping from static images to complicated activity concepts. In this paper, we propose a novel "Align before Adapt" (ALT) paradigm. Prior to adapting to video representation learning, we exploit the entity-to-region alignments for each frame. The alignments are fulfilled by matching the region-aware image embeddings to an offline-constructed text corpus. With the aligned entities, we feed their text embeddings to a transformer-based video adapter as the queries, which can help extract the semantics of the most important entities from a video to a vector. This paradigm reuses the visual-language alignment of VLP during adaptation and tries to explain an action by the underlying entities. This helps understand actions by bridging the gap with complex activity semantics, particularly when facing unfamiliar or unseen categories. ALT demonstrates competitive performance while maintaining remarkably low computational costs. In fully supervised experiments, it achieves 88.1% top-1 accuracy on Kinetics-400 with only 4947 GFLOPs. Moreover, ALT outperforms the previous state-of-the-art methods in both zero-shot and few-shot experiments, emphasizing its superior generalizability across various learning scenarios.

翻译：大规模视觉-语言预训练模型在各种视频任务中取得了显著成功。然而，现有方法大多遵循“先适应后对齐”范式，即先调整预训练图像编码器以建模视频级表示，再利用动作标签的独热编码或文本嵌入进行监督。这种范式忽视了从静态图像到复杂活动概念的映射挑战。本文提出一种新颖的“对齐前适应”（ALT）范式。在适应视频表示学习之前，我们为每一帧探索实体-区域对齐。这种对齐通过将区域感知图像嵌入与离线构建的文本语料库匹配来实现。利用对齐后的实体，我们将其文本嵌入输入到基于Transformer的视频适配器作为查询，从而帮助从视频中提取最重要实体的语义，并将其转化为向量。该范式在适应过程中重用了视觉-语言对齐，并试图通过底层实体解释动作。这有助于弥合与复杂活动语义之间的差距来理解动作，尤其是在面对不熟悉或未见类别时。ALT在保持极低计算成本的同时展现了竞争性性能。在全监督实验中，它在Kinetics-400上仅需4947 GFLOPs即可达到88.1%的top-1准确率。此外，ALT在零样本和小样本实验中均超越了先前最先进方法，突显了其在各种学习场景中的卓越泛化能力。