We propose Video Localized Narratives, a new form of multimodal video annotations connecting vision and language. In the original Localized Narratives, annotators speak and move their mouse simultaneously on an image, thus grounding each word with a mouse trace segment. However, this is challenging on a video. Our new protocol empowers annotators to tell the story of a video with Localized Narratives, capturing even complex events involving multiple actors interacting with each other and with several passive objects. We annotated 20k videos of the OVIS, UVO, and Oops datasets, totalling 1.7M words. Based on this data, we also construct new benchmarks for the video narrative grounding and video question answering tasks, and provide reference results from strong baseline models. Our annotations are available at https://google.github.io/video-localized-narratives/.

翻译：我们提出了视频本地化叙事（Video Localized Narratives），这是一种连接视觉与语言的新型多模态视频标注形式。在原始的本地化叙事中，标注者同时说话并在图像上移动鼠标，从而将每个词语与鼠标轨迹片段相关联。然而，这一方法在视频上具有挑战性。我们的新流程使标注者能够通过本地化叙事讲述视频故事，甚至能捕捉涉及多个行为体相互交互以及与若干被动对象互动的复杂事件。我们对OVIS、UVO和Oops数据集的2万段视频进行了标注，总词汇量达170万。基于该数据，我们还构建了视频叙事定位和视频问答任务的新基准，并提供了强基线模型的参考结果。我们的标注数据可于https://google.github.io/video-localized-narratives/获取。