A key challenge with procedure planning in instructional videos lies in how to handle a large decision space consisting of a multitude of action types that belong to various tasks. To understand real-world video content, an AI agent must proficiently discern these action types (e.g., pour milk, pour water, open lid, close lid, etc.) based on brief visual observation. Moreover, it must adeptly capture the intricate semantic relation of the action types and task goals, along with the variable action sequences. Recently, notable progress has been made via the integration of diffusion models and visual representation learning to address the challenge. However, existing models employ rudimentary mechanisms to utilize task information to manage the decision space. To overcome this limitation, we introduce a simple yet effective enhancement - a masked diffusion model. The introduced mask acts akin to a task-oriented attention filter, enabling the diffusion/denoising process to concentrate on a subset of action types. Furthermore, to bolster the accuracy of task classification, we harness more potent visual representation learning techniques. In particular, we learn a joint visual-text embedding, where a text embedding is generated by prompting a pre-trained vision-language model to focus on human actions. We evaluate the method on three public datasets and achieve state-of-the-art performance on multiple metrics. Code is available at https://github.com/ffzzy840304/Masked-PDPP.

翻译：教学视频中程序规划的核心挑战在于如何应对由属于不同任务的大量动作类型构成的大规模决策空间。为理解真实世界视频内容，AI智能体必须基于简短视觉观察熟练辨别这些动作类型（如倒牛奶、倒水、开盖、关盖等），并精准捕捉动作类型与任务目标间的复杂语义关系及可变动作序列。近年来，通过融合扩散模型与视觉表征学习在应对该挑战上取得显著进展，但现有模型仍采用基础机制利用任务信息管理决策空间。为突破此局限，我们提出一种简洁而有效的增强方法——掩膜扩散模型。该掩膜类似于任务导向的注意力过滤器，使扩散/去噪过程聚焦于动作类型的子集。此外，为提升任务分类准确性，我们采用更强大的视觉表征学习技术，具体通过构建联合视觉-文本嵌入，利用提示预训练视觉语言模型聚焦人类行为生成文本嵌入。该方法在三个公开数据集上实现评估，并在多项指标上达到最优性能。代码已开源：https://github.com/ffzzy840304/Masked-PDPP。