Early weakly supervised video grounding (WSVG) methods often struggle with incomplete boundary detection due to the absence of temporal boundary annotations. To bridge the gap between video-level and boundary-level annotation, explicit-supervision methods, i.e., generating pseudo-temporal boundaries for training, have achieved great success. However, data augmentations in these methods might disrupt critical temporal information, yielding poor pseudo boundaries. In this paper, we propose a new perspective that maintains the integrity of the original temporal content while introducing more valuable information for expanding the incomplete boundaries. To this end, we propose EtC (Expand then Clarify), first use the additional information to expand the initial incomplete pseudo boundaries, and subsequently refine these expanded ones to achieve precise boundaries. Motivated by video continuity, i.e., visual similarity across adjacent frames, we use powerful multimodal large language models (MLLMs) to annotate each frame within initial pseudo boundaries, yielding more comprehensive descriptions for expanded boundaries. To further clarify the noise of expanded boundaries, we combine mutual learning with a tailored proposal-level contrastive objective to use a learnable approach to harmonize a balance between incomplete yet clean (initial) and comprehensive yet noisy (expanded) boundaries for more precise ones. Experiments demonstrate the superiority of our method on two challenging WSVG datasets.
翻译:早期弱监督视频定位方法因缺乏时间边界标注,常难以完整检测边界。为弥合视频级与边界级标注之间的差距,显式监督方法(即生成伪时间边界用于训练)已取得显著成效。然而,此类方法中的数据增强可能破坏关键时间信息,导致伪边界质量不佳。本文提出一种新视角:在保持原始时间内容完整性的同时,引入更具价值的信息以扩展不完整边界。为此,我们提出EtC(先扩展后澄清)方法:首先利用附加信息扩展初始不完整伪边界,随后对扩展后的边界进行精炼以获取精确边界。受视频连续性(即相邻帧的视觉相似性)启发,我们利用强大的多模态大语言模型对初始伪边界内的每一帧进行标注,为扩展边界生成更全面的描述。为进一步澄清扩展边界中的噪声,我们将互学习与定制化的提议级对比目标相结合,通过可学习方法协调不完整但干净(初始)与全面但有噪声(扩展)两类边界之间的平衡,从而获得更精准的边界。实验结果表明,本方法在两个具有挑战性的弱监督视频定位数据集上均展现出优越性能。