We propose a novel approach to the action segmentation task for long, untrimmed videos, based on solving an optimal transport problem. By encoding a temporal consistency prior into a Gromov-Wasserstein problem, we are able to decode a temporally consistent segmentation from a noisy affinity/matching cost matrix between video frames and action classes. Unlike previous approaches, our method does not require knowing the action order for a video to attain temporal consistency. Furthermore, our resulting (fused) Gromov-Wasserstein problem can be efficiently solved on GPUs using a few iterations of projected mirror descent. We demonstrate the effectiveness of our method in an unsupervised learning setting, where our method is used to generate pseudo-labels for self-training. We evaluate our segmentation approach and unsupervised learning pipeline on the Breakfast, 50-Salads, YouTube Instructions and Desktop Assembly datasets, yielding state-of-the-art results for the unsupervised video action segmentation task.

翻译：我们提出了一种新颖的方法来解决长时未裁剪视频的动作分割任务，该方法基于求解最优传输问题。通过将时间一致性先验编码到Gromov-Wasserstein问题中，我们能够从视频帧与动作类别之间的噪声亲和/匹配成本矩阵中解码出时间一致的分割结果。与以往方法不同，我们的方法无需知晓视频中的动作顺序即可实现时间一致性。此外，我们得到的（融合）Gromov-Wasserstein问题可通过仅几次投影镜像下降迭代在GPU上高效求解。我们在无监督学习设置中展示了该方法的有效性，即利用其生成用于自训练的伪标签。我们在Breakfast、50-Salads、YouTube Instructions和Desktop Assembly数据集上评估了我们的分割方法及无监督学习流程，在无监督视频动作分割任务中取得了最先进的结果。