In this paper we present an approach for localizing steps of procedural activities in narrated how-to videos. To deal with the scarcity of labeled data at scale, we source the step descriptions from a language knowledge base (wikiHow) containing instructional articles for a large variety of procedural tasks. Without any form of manual supervision, our model learns to temporally ground the steps of procedural articles in how-to videos by matching three modalities: frames, narrations, and step descriptions. Specifically, our method aligns steps to video by fusing information from two distinct pathways: i) {\em direct} alignment of step descriptions to frames, ii) {\em indirect} alignment obtained by composing steps-to-narrations with narrations-to-video correspondences. Notably, our approach performs global temporal grounding of all steps in an article at once by exploiting order information, and is trained with step pseudo-labels which are iteratively refined and aggressively filtered. In order to validate our model we introduce a new evaluation benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of HowTo100M\footnote{A test server is accessible at \url{https://eval.ai/web/challenges/challenge-page/2082}.} with steps sourced from wikiHow articles. Experiments on this benchmark as well as zero-shot evaluations on CrossTask demonstrate that our multi-modality alignment yields dramatic gains over several baselines and prior works. Finally, we show that our inner module for matching narration-to-video outperforms by a large margin the state of the art on the HTM-Align narration-video alignment benchmark.

翻译：在本文中，我们提出了一种在带叙述的“如何做”视频中定位程序性活动步骤的方法。为了应对大规模标注数据稀缺的问题，我们从语言知识库（wikiHow）中获取步骤描述，该知识库包含大量程序性任务的教学文章。在没有任何人工标注的情况下，我们的模型通过匹配三种模态：帧、叙述和步骤描述，学习在时间上定位“如何做”视频中程序性文章的步骤。具体来说，我们的方法通过融合两条不同路径的信息将步骤与视频对齐：i) 步骤描述与帧的直接对齐，ii) 通过组合步骤到叙述和叙述到视频的对应关系获得的间接对齐。值得注意的是，我们的方法通过利用顺序信息，一次性实现文章中所有步骤的全局时间定位，并使用逐步精炼和严格过滤的步骤伪标签进行训练。为了验证我们的模型，我们引入了一个新的评估基准——HT-Step，该基准通过手动标注HowTo100M\footnote{测试服务器可通过\url{https://eval.ai/web/challenges/challenge-page/2082}访问。}中124小时子集，并附上来自wikiHow文章的步骤。在该基准上的实验以及在CrossTask上的零样本评估表明，我们的多模态对齐方法相较于多个基线和先前工作取得了显著提升。最后，我们展示了用于匹配叙述到视频的内部模块在HTM-Align叙述-视频对齐基准上大幅超越了现有技术水平。