We present ActionDiffusion -- a novel diffusion model for procedure planning in instructional videos that is the first to take temporal inter-dependencies between actions into account in a diffusion model for procedure planning. This approach is in stark contrast to existing methods that fail to exploit the rich information content available in the particular order in which actions are performed. Our method unifies the learning of temporal dependencies between actions and denoising of the action plan in the diffusion process by projecting the action information into the noise space. This is achieved 1) by adding action embeddings in the noise masks in the noise-adding phase and 2) by introducing an attention mechanism in the noise prediction network to learn the correlations between different action steps. We report extensive experiments on three instructional video benchmark datasets (CrossTask, Coin, and NIV) and show that our method outperforms previous state-of-the-art methods on all metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset. We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning.

翻译：我们提出ActionDiffusion——一种用于教学视频中过程规划的新型扩散模型，这是首个在过程规划扩散模型中考虑动作之间时间相互依赖关系的方法。该方法与现有方法形成鲜明对比，后者未能利用动作执行特定顺序中蕴含的丰富信息。我们的方法通过将动作信息投影到噪声空间，统一了扩散过程中动作间时间依赖性的学习与动作计划的去噪过程。这通过以下两种方式实现：1) 在噪声添加阶段将动作嵌入添加到噪声掩码中；2) 在噪声预测网络中引入注意力机制以学习不同动作步骤之间的相关性。我们在三个教学视频基准数据集（CrossTask、Coin和NIV）上进行了广泛实验，结果表明，我们的方法在CrossTask和NIV数据集上的所有指标以及Coin数据集上除准确率外的所有指标均优于先前最先进的方法。实验证明，通过将动作嵌入添加到噪声掩码中，扩散模型能够更好地学习动作时间依赖性，并提升过程规划的性能。