Real-world tasks consist of multiple inter-dependent subtasks (e.g., a dirty pan needs to be washed before it can be used for cooking). In this work, we aim to model the causal dependencies between such subtasks from instructional videos describing the task. This is a challenging problem since complete information about the world is often inaccessible from videos, which demands robust learning mechanisms to understand the causal structure of events. We present Multimodal Subtask Graph Generation (MSG2), an approach that constructs a Subtask Graph defining the dependency between a task's subtasks relevant to a task from noisy web videos. Graphs generated by our multimodal approach are closer to human-annotated graphs compared to prior approaches. MSG2 further performs the downstream task of next subtask prediction 85% and 30% more accurately than recent video transformer models in the ProceL and CrossTask datasets, respectively.

翻译：现实世界的任务由多个相互依赖的子任务组成（例如，脏锅在使用前需要先清洗）。本研究旨在从描述任务的教学视频中建模此类子任务之间的因果依赖关系。由于视频中往往无法获取完整的世界信息，这需要鲁棒的学习机制来理解事件的因果结构，因此这是一个具有挑战性的问题。我们提出多模态子任务图生成（MSG2）方法，该方法能从含噪的网络视频中构建定义任务子任务之间依赖关系的子任务图。与先前方法相比，我们的多模态方法生成的图更接近人工标注的图。在ProceL和CrossTask数据集上，MSG2在下游任务——下一子任务预测中的准确率分别比最新的视频Transformer模型高出85%和30%。