The abundance of instructional videos and their narrations over the Internet offers an exciting avenue for understanding procedural activities. In this work, we propose to learn video representation that encodes both action steps and their temporal ordering, based on a large-scale dataset of web instructional videos and their narrations, without using human annotations. Our method jointly learns a video representation to encode individual step concepts, and a deep probabilistic model to capture both temporal dependencies and immense individual variations in the step ordering. We empirically demonstrate that learning temporal ordering not only enables new capabilities for procedure reasoning, but also reinforces the recognition of individual steps. Our model significantly advances the state-of-the-art results on step classification (+2.8% / +3.3% on COIN / EPIC-Kitchens) and step forecasting (+7.4% on COIN). Moreover, our model attains promising results in zero-shot inference for step classification and forecasting, as well as in predicting diverse and plausible steps for incomplete procedures. Our code is available at https://github.com/facebookresearch/ProcedureVRL.

翻译：互联网上丰富的教学视频及其旁白为理解过程性活动提供了令人兴奋的途径。本文提出在无需人工标注的条件下，基于大规模网络教学视频及其旁白数据集，学习能够编码动作步骤及其时间顺序的视频表示。我们的方法联合学习视频表示以编码单个步骤概念，并构建深度概率模型以捕获时间依赖关系及步骤排序中巨大的个体差异。实验证明，学习时间顺序不仅能赋予过程推理新能力，还能强化对单个步骤的识别。我们的模型在步骤分类（COIN数据集提升+2.8%，EPIC-Kitchens数据集提升+3.3%）和步骤预测（COIN数据集提升+7.4%）任务上显著超越现有最优方法。此外，该模型在步骤分类与预测的零样本推理任务，以及为不完整过程预测多样化且合理的步骤方面均取得优异表现。代码已开源：https://github.com/facebookresearch/ProcedureVRL。