Video Temporal Grounding is to identify specific moments or highlights from a video corresponding to textual descriptions. Typical approaches in temporal grounding treat all video clips equally during the encoding process regardless of their semantic relevance with the text query. Therefore, we propose Correlation-Guided DEtection TRansformer(CG-DETR), exploring to provide clues for query-associated video clips within the cross-modal attention. First, we design an adaptive cross-attention with dummy tokens. Dummy tokens conditioned by text query take portions of the attention weights, preventing irrelevant video clips from being represented by the text query. Yet, not all words equally inherit the text query's correlation to video clips. Thus, we further guide the cross-attention map by inferring the fine-grained correlation between video clips and words. We enable this by learning a joint embedding space for high-level concepts, i.e., moment and sentence level, and inferring the clip-word correlation. Lastly, we exploit the moment-specific characteristics and combine them with the context of each video to form a moment-adaptive saliency detector. By exploiting the degrees of text engagement in each video clip, it precisely measures the highlightness of each clip. CG-DETR achieves state-of-the-art results on various benchmarks for temporal grounding.

翻译：视频时序定位旨在从视频中识别与文本描述相对应的特定时刻或高光片段。现有的大多数时序定位方法在编码过程中对所有视频片段一视同仁，忽略了它们与文本查询的语义相关性。为此，本文提出相关性引导的检测变换器（CG-DETR），探索在跨模态注意力中为与查询相关的视频片段提供线索。首先，我们设计了一种带虚拟令牌的自适应交叉注意力机制。由文本查询调节的虚拟令牌占据部分注意力权重，从而阻止不相关的视频片段被文本查询表示。然而，并非所有词语都同等继承文本查询与视频片段的相关性。因此，我们通过推断视频片段与词语之间的细粒度相关性，进一步引导交叉注意力图。具体而言，我们通过为高层概念（即时刻和句子层面）学习联合嵌入空间，并推断片段-词语相关性来实现这一目标。最后，我们利用时刻特定的特征，将其与每个视频的上下文相结合，形成时刻自适应显著性检测器。通过利用每个视频片段中文本参与程度的度量，该检测器精确衡量各片段的高亮度。CG-DETR在多个时序定位基准上取得了最先进的结果。