Joint video-language learning has received increasing attention in recent years. However, existing works mainly focus on single or multiple trimmed video clips (events), which makes human-annotated event boundaries necessary during inference. To break away from the ties, we propose a grounded vision-language learning framework for untrimmed videos, which automatically detects informative events and effectively excavates the alignments between multi-sentence descriptions and corresponding event segments. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments, i.e., text-to-event grounding (TEG) and event-to-text generation (ETG). TEG learns to adaptively ground the possible event proposals given a set of sentences by estimating the cross-modal distance in a joint semantic space. Meanwhile, ETG aims to reconstruct (generate) the matched texts given event proposals, encouraging the event representation to retain meaningful semantic information. To encourage accurate label assignment between the event set and the text set, we propose a novel semantic-aware cost to mitigate the sub-optimal matching results caused by ambiguous boundary annotations. Our framework is easily extensible to tasks covering visually-grounded language understanding and generation. We achieve state-of-the-art dense video captioning performance on ActivityNet Captions, YouCook2 and YouMakeup, and competitive performance on several other language generation and understanding tasks. Our method also achieved 1st place in both the MTVG and MDVC tasks of the PIC 4th Challenge. Our code is publicly available at https://github.com/zjr2000/GVL.

翻译：近年来，视频-语言联合学习受到越来越多的关注。然而，现有研究主要聚焦于单一或多个修剪后的视频片段（事件），导致推理过程中必须依赖人工标注的事件边界。为摆脱这一束缚，我们提出了一种针对未修剪视频的接地式视觉-语言学习框架，该框架能自动检测信息性事件，并有效挖掘多句描述与相应事件片段之间的对齐关系。与粗粒度的视频-语言对齐不同，我们设计了两种双重前置任务以促进细粒度片段级对齐：文本-事件接地（TEG）与事件-文本生成（ETG）。TEG通过估计联合语义空间中的跨模态距离，自适应地将给定句子集与可能的事件提案进行接地；ETG则旨在根据事件提案重建（生成）匹配文本，促使事件表征保留有意义的语义信息。为实现事件集与文本集间准确的标签分配，我们提出了一种新型语义感知代价函数，以缓解由模糊边界标注导致的次优匹配结果。该框架可轻松扩展到涵盖视觉基础的语言理解与生成任务。我们在ActivityNet Captions、YouCook2和YouMakeup数据集上实现了最优的稠密视频描述性能，并在多项其他语言生成与理解任务中取得了具有竞争力的结果。我们的方法还在PIC第四届挑战赛的MTVG和MDVC任务中均获得了第一名。代码已公开于https://github.com/zjr2000/GVL。