The correlation between the vision and text is essential for video moment retrieval (VMR), however, existing methods heavily rely on separate pre-training feature extractors for visual and textual understanding. Without sufficient temporal boundary annotations, it is non-trivial to learn universal video-text alignments. In this work, we explore multi-modal correlations derived from large-scale image-text data to facilitate generalisable VMR. To address the limitations of image-text pre-training models on capturing the video changes, we propose a generic method, referred to as Visual-Dynamic Injection (VDI), to empower the model's understanding of video moments. Whilst existing VMR methods are focusing on building temporal-aware video features, being aware of the text descriptions about the temporal changes is also critical but originally overlooked in pre-training by matching static images with sentences. Therefore, we extract visual context and spatial dynamic information from video frames and explicitly enforce their alignments with the phrases describing video changes (e.g. verb). By doing so, the potentially relevant visual and motion patterns in videos are encoded in the corresponding text embeddings (injected) so to enable more accurate video-text alignments. We conduct extensive experiments on two VMR benchmark datasets (Charades-STA and ActivityNet-Captions) and achieve state-of-the-art performances. Especially, VDI yields notable advantages when being tested on the out-of-distribution splits where the testing samples involve novel scenes and vocabulary.

翻译：视觉与文本之间的关联对于视频段落检索至关重要，然而现有方法严重依赖针对视觉与文本理解的独立预训练特征提取器。在缺乏足够时间边界标注的情况下，学习通用视频-文本对齐并非易事。本文探索利用大规模图像-文本数据中推导出的多模态关联，以促进可泛化的视频段落检索。为克服图像-文本预训练模型在捕捉视频动态变化方面的局限性，我们提出一种名为视觉动态注入的通用方法，以增强模型对视频段落的理解能力。现有视频段落检索方法主要聚焦于构建时间感知的视频特征，而预训练过程中通过静态图像与句子匹配原本忽略了关于时间变化的文本描述——这一感知同样关键。为此，我们从视频帧中提取视觉上下文与空间动态信息，并强制其与描述视频变化（如动词）的短语进行对齐。通过此操作，视频中潜在相关的视觉与运动模式被编码到对应文本嵌入中（注入），从而实现更精准的视频-文本对齐。我们在两个视频段落检索基准数据集（Charades-STA与ActivityNet-Captions）上开展广泛实验，取得当前最佳性能。尤其当在测试样本涉及新场景与新词汇的分布外划分上进行评估时，VDI展现出显著优势。