In a video, an object usually appears as the trajectory, i.e., it spans over a few spatial but longer temporal patches, that contains abundant spatiotemporal contexts. However, modern Video-Language BERTs (VDL-BERTs) neglect this trajectory characteristic that they usually follow image-language BERTs (IL-BERTs) to deploy the patch-to-word (P2W) attention that may over-exploit trivial spatial contexts and neglect significant temporal contexts. To amend this, we propose a novel TW-BERT to learn Trajectory-Word alignment by a newly designed trajectory-to-word (T2W) attention for solving video-language tasks. Moreover, previous VDL-BERTs usually uniformly sample a few frames into the model while different trajectories have diverse graininess, i.e., some trajectories span longer frames and some span shorter, and using a few frames will lose certain useful temporal contexts. However, simply sampling more frames will also make pre-training infeasible due to the largely increased training burdens. To alleviate the problem, during the fine-tuning stage, we insert a novel Hierarchical Frame-Selector (HFS) module into the video encoder. HFS gradually selects the suitable frames conditioned on the text context for the later cross-modal encoder to learn better trajectory-word alignments. By the proposed T2W attention and HFS, our TW-BERT achieves SOTA performances on text-to-video retrieval tasks, and comparable performances on video question-answering tasks with some VDL-BERTs trained on much more data. The code will be available in the supplementary material.

翻译：在视频中，物体通常以轨迹形式出现，即跨越少量空间区域但较长时间片段，包含丰富的时空上下文信息。然而，现代视频语言BERT模型（VDL-BERT）忽略了这一轨迹特性，它们通常沿用图像语言BERT模型（IL-BERT）的片段-词语（P2W）注意力机制，这可能过度挖掘琐碎的空间上下文而忽略了重要时序上下文。为此，我们提出新颖的TW-BERT模型，通过新设计的轨迹-词语（T2W）注意力机制学习轨迹-词语对齐，以解决视频语言任务。此外，以往VDL-BERT通常采用均匀采样少量帧输入模型，而不同轨迹具有不同粒度——部分轨迹跨越较长帧段，部分轨迹较短，采用少量帧会丢失部分有用时序上下文。但简单增加采样帧数会导致训练负担大幅增加，使得预训练不可行。为缓解此问题，我们在微调阶段在视频编码器中插入新颖的分层帧选择器（HFS）模块。HFS基于文本上下文逐步选择合适帧，供后续跨模态编码器学习更优的轨迹-词语对齐。通过所提T2W注意力和HFS，我们的TW-BERT在文本到视频检索任务上取得最优性能，在视频问答任务上达到与部分使用更多数据训练的VDL-BERT相当的性能。代码将在补充材料中提供。