Instructional videos are an important resource to learn procedural tasks from human demonstrations. However, the instruction steps in such videos are typically short and sparse, with most of the video being irrelevant to the procedure. This motivates the need to temporally localize the instruction steps in such videos, i.e. the task called key-step localization. Traditional methods for key-step localization require video-level human annotations and thus do not scale to large datasets. In this work, we tackle the problem with no human supervision and introduce StepFormer, a self-supervised model that discovers and localizes instruction steps in a video. StepFormer is a transformer decoder that attends to the video with learnable queries, and produces a sequence of slots capturing the key-steps in the video. We train our system on a large dataset of instructional videos, using their automatically-generated subtitles as the only source of supervision. In particular, we supervise our system with a sequence of text narrations using an order-aware loss function that filters out irrelevant phrases. We show that our model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization by a large margin on three challenging benchmarks. Moreover, our model demonstrates an emergent property to solve zero-shot multi-step localization and outperforms all relevant baselines at this task.

翻译：教学视频是从人类示范中学习程序性任务的重要资源。然而，此类视频中的指令步骤通常短且稀疏，大部分视频内容与流程无关。这促使我们需要对这些视频中的指令步骤进行时间定位，即被称为关键步骤定位的任务。传统的关键步骤定位方法需要视频级人工标注，因此无法扩展到大规模数据集。在本文中，我们无需任何人工监督来解决该问题，并提出了StepFormer——一种自监督模型，能够发现并定位视频中的指令步骤。StepFormer是一种Transformer解码器，通过可学习查询关注视频内容，并生成捕获视频中关键步骤的槽位序列。我们在大规模教学视频数据集上训练该系统，仅使用自动生成的字幕作为监督来源。具体而言，我们通过文本叙述序列并结合顺序感知损失函数对系统进行监督，该损失函数可过滤掉无关短语。实验表明，我们的模型在三个具有挑战性的基准测试中，其步骤检测与定位性能大幅超越了所有先前的无监督和弱监督方法。此外，该模型展现出解决零样本多步骤定位的新兴能力，并在该任务上优于所有相关基线方法。